[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 13:17:37.835705 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:39733
I20240426 13:17:37.837103 23796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 13:17:37.838186 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:37.854764 23805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:37.854813 23802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:37.854805 23803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:37.856245 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:38.949949 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:38.950196 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:38.950349 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137458950334 us; error 0 us; skew 500 ppm
I20240426 13:17:38.951161 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:38.956547 23796 webserver.cc:460] Webserver started at http://127.23.61.62:37521/ using document root <none> and password file <none>
I20240426 13:17:38.957419 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:38.957612 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:38.957986 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:38.964125 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "5631be9661c5413097ecac9e9e7835e5"
format_stamp: "Formatted at 2024-04-26 13:17:38 on dist-test-slave-jx77"
I20240426 13:17:38.971596 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240426 13:17:38.976673 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:38.977733 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240426 13:17:38.978039 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "5631be9661c5413097ecac9e9e7835e5"
format_stamp: "Formatted at 2024-04-26 13:17:38 on dist-test-slave-jx77"
I20240426 13:17:38.978356 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.042387 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:17:39.043839 23796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 13:17:39.044274 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.112521 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:39733
I20240426 13:17:39.112605 23862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:39733 every 8 connection(s)
I20240426 13:17:39.119544 23863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:39.142756 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Bootstrap starting.
I20240426 13:17:39.148125 23863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:39.149859 23863 log.cc:824] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Log is configured to *not* fsync() on all Append() calls
I20240426 13:17:39.154168 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: No bootstrap required, opened a new log
I20240426 13:17:39.175776 23863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.176342 23863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:39.176538 23863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5631be9661c5413097ecac9e9e7835e5, State: Initialized, Role: FOLLOWER
I20240426 13:17:39.177264 23863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.177742 23863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:17:39.177973 23863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:17:39.178249 23863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:39.181891 23863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.182507 23863 leader_election.cc:302] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5631be9661c5413097ecac9e9e7835e5; no voters:
I20240426 13:17:39.184083 23863 leader_election.cc:288] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 13:17:39.184415 23866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:39.186975 23866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5631be9661c5413097ecac9e9e7835e5, State: Running, Role: LEADER
I20240426 13:17:39.187777 23866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER }
I20240426 13:17:39.188818 23863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:17:39.198256 23867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5631be9661c5413097ecac9e9e7835e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER } }
I20240426 13:17:39.198633 23868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5631be9661c5413097ecac9e9e7835e5. Latest consensus state: current_term: 1 leader_uuid: "5631be9661c5413097ecac9e9e7835e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5631be9661c5413097ecac9e9e7835e5" member_type: VOTER } }
I20240426 13:17:39.199178 23868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: This master's current role is: LEADER
I20240426 13:17:39.199178 23867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [sys.catalog]: This master's current role is: LEADER
I20240426 13:17:39.202414 23876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:17:39.213723 23876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:17:39.220765 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:17:39.230899 23876 catalog_manager.cc:1347] Generated new cluster ID: 57169e0a2011421f9d4f00a8bad18da1
I20240426 13:17:39.231202 23876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:17:39.246026 23876 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:17:39.248092 23876 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:17:39.270504 23876 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: Generated new TSK 0
I20240426 13:17:39.271649 23876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:17:39.287178 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.295161 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.296100 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.302160 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.303112 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.303745 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.303943 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.304131 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459304115 us; error 0 us; skew 500 ppm
I20240426 13:17:39.304713 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.308569 23796 webserver.cc:460] Webserver started at http://127.23.61.1:44903/ using document root <none> and password file <none>
I20240426 13:17:39.309060 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.309281 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.309554 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.310653 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.316445 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240426 13:17:39.320454 23893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.321280 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240426 13:17:39.321568 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.321852 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.361940 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:17:39.363134 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.365703 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.370214 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.370432 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.370733 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.370885 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.472268 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:44541
I20240426 13:17:39.472393 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:44541 every 8 connection(s)
I20240426 13:17:39.478868 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.491952 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.497804 23961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:17:39.508445 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.513607 23956 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.514098 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.514106 23956 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.514999 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.515209 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.515383 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459515368 us; error 0 us; skew 500 ppm
I20240426 13:17:39.515380 23956 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
I20240426 13:17:39.516150 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.519349 23828 ts_manager.cc:194] Registered new tserver with Master: cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541)
I20240426 13:17:39.519467 23796 webserver.cc:460] Webserver started at http://127.23.61.2:34327/ using document root <none> and password file <none>
I20240426 13:17:39.520174 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.520429 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.520774 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.522296 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "decbe9ee3d0a47bab14b976ecaf12843"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.522325 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54958
I20240426 13:17:39.528173 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240426 13:17:39.532128 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.532915 23796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240426 13:17:39.533179 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "decbe9ee3d0a47bab14b976ecaf12843"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.533473 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.558553 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:17:39.559794 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.562222 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.565181 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.565358 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.565609 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.565748 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.661548 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:35101
I20240426 13:17:39.661687 24030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:35101 every 8 connection(s)
I20240426 13:17:39.669883 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:17:39.678973 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.682168 24031 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.682664 24031 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.683866 24031 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
W20240426 13:17:39.686048 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.686450 23828 ts_manager.cc:194] Registered new tserver with Master: decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:39.688331 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54970
W20240426 13:17:39.688760 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:17:39.693871 23796 server_base.cc:1034] running on GCE node
I20240426 13:17:39.694659 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:17:39.694859 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:17:39.695030 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137459695011 us; error 0 us; skew 500 ppm
I20240426 13:17:39.695574 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:17:39.698576 23796 webserver.cc:460] Webserver started at http://127.23.61.3:40183/ using document root <none> and password file <none>
I20240426 13:17:39.699074 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:17:39.699277 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:17:39.699532 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:17:39.700862 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "20b204c02fe14d9eb754d5a169d43996"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.705523 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240426 13:17:39.708752 24042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.709534 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240426 13:17:39.709789 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "20b204c02fe14d9eb754d5a169d43996"
format_stamp: "Formatted at 2024-04-26 13:17:39 on dist-test-slave-jx77"
I20240426 13:17:39.710047 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:17:39.747560 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:17:39.748759 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:17:39.750208 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:17:39.753605 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:17:39.753803 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.754062 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:17:39.754205 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:17:39.847970 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:45215
I20240426 13:17:39.848059 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:45215 every 8 connection(s)
I20240426 13:17:39.864207 24105 heartbeater.cc:347] Connected to a master server at 127.23.61.62:39733
I20240426 13:17:39.864549 24105 heartbeater.cc:464] Registering TS with master...
I20240426 13:17:39.865213 24105 heartbeater.cc:511] Master 127.23.61.62:39733 requested a full tablet report, sending...
I20240426 13:17:39.867152 23828 ts_manager.cc:194] Registered new tserver with Master: 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:39.868008 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015310463s
I20240426 13:17:39.868956 23828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54980
I20240426 13:17:39.872473 23796 test_util.cc:268] Using random seed: -1167830334
I20240426 13:17:39.921759 23828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54988:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 13:17:39.925360 23828 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:17:40.029588 23920 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.030342 23919 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.031277 23920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.031996 23919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.034143 23918 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.035632 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.033249 23921 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.041544 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.046172 23917 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.047506 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.051560 23916 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.054275 23916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.059098 23915 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.061138 23915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.070240 24070 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.070798 24069 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.073320 24070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.084362 24069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.076122 24068 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.086189 24067 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.089030 24065 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.095767 24068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.097096 24067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.081506 24066 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.095208 23914 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.096175 24064 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.111622 23996 tablet_service.cc:1467] Processing CreateTablet for tablet c9fc30fbb8de42d58ac588a5f723f4fe (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240426 13:17:40.107662 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.110447 23914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.119814 23996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9fc30fbb8de42d58ac588a5f723f4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.111455 24063 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.126462 23995 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc471e293e547cca28b0aecdccbb30d (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240426 13:17:40.118155 23913 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.126873 24066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.139787 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc471e293e547cca28b0aecdccbb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.142683 24062 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.137437 23994 tablet_service.cc:1467] Processing CreateTablet for tablet 52fc238a1b814a388b781f80b802ba71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240426 13:17:40.143545 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.132019 23993 tablet_service.cc:1467] Processing CreateTablet for tablet 53b6922ff368484e93a5442573ad8b85 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240426 13:17:40.149688 24123 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.156499 23913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.149662 23992 tablet_service.cc:1467] Processing CreateTablet for tablet 90f60b45962d422292f2f367e49d0376 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240426 13:17:40.162138 23989 tablet_service.cc:1467] Processing CreateTablet for tablet eebc1d91fea64133989f26f880fdec71 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240426 13:17:40.163507 23989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.166716 23988 tablet_service.cc:1467] Processing CreateTablet for tablet 6ada48e64e5a4d36b5186827ace67fb9 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240426 13:17:40.167678 23990 tablet_service.cc:1467] Processing CreateTablet for tablet 734eaab0271041ecaa0e13f446948d2e (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240426 13:17:40.171777 23994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52fc238a1b814a388b781f80b802ba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.173537 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.170680 24062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ada48e64e5a4d36b5186827ace67fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.177692 23991 tablet_service.cc:1467] Processing CreateTablet for tablet 27e6fe9ec33c40f8aa98d61394752d08 (DEFAULT_TABLE table=test-workload [id=c235339495c0449c8e79b61762cc1dbc]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240426 13:17:40.189545 23992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f60b45962d422292f2f367e49d0376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.202613 23991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e6fe9ec33c40f8aa98d61394752d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.204478 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734eaab0271041ecaa0e13f446948d2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:17:40.213791 23875 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:17:40.217510 23993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b6922ff368484e93a5442573ad8b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.221472 24063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebc1d91fea64133989f26f880fdec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:17:40.318614 24123 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.349287 24124 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.351624 24125 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.367540 24123 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.368764 24124 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.370522 24123 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.236s user 0.015s sys 0.004s
I20240426 13:17:40.374158 24123 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.374966 24123 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.375272 24123 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.375663 24125 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.375979 24123 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.389058 23956 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.389614 24123 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.018s user 0.005s sys 0.013s
I20240426 13:17:40.390476 24124 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.390763 24123 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.390906 24124 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.007s
I20240426 13:17:40.396493 24123 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.403947 24124 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.404999 24124 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.405375 24124 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.409324 24125 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.409829 24125 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.023s
I20240426 13:17:40.410676 24124 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.412696 24125 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.413612 24125 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.413952 24125 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.414752 24125 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.422921 24123 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.423394 24123 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.011s
I20240426 13:17:40.423768 24125 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20240426 13:17:40.424257 24124 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.033s user 0.007s sys 0.003s
I20240426 13:17:40.424533 24031 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.424857 24125 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.428650 24105 heartbeater.cc:503] Master 127.23.61.62:39733 was elected leader, sending a full tablet report...
I20240426 13:17:40.428897 24124 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.432303 24132 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.432848 24132 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.436134 24123 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.436928 24123 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.437223 24123 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.438850 24125 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.439136 24123 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.441327 24124 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.446094 24132 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.450212 24123 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.027s user 0.010s sys 0.002s
I20240426 13:17:40.457346 24124 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.457775 24124 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.005s
I20240426 13:17:40.460738 24125 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.462314 24125 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.009s
I20240426 13:17:40.465607 24123 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.460364 24124 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.468299 24124 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.468461 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.469094 24124 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.470989 24124 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
W20240426 13:17:40.472391 24045 leader_election.cc:341] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.473570 24124 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20240426 13:17:40.473081 24125 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.474434 24124 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.474706 24125 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.476006 24125 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.477352 24123 tablet_bootstrap.cc:654] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
W20240426 13:17:40.480491 23957 tablet.cc:2369] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 13:17:40.477552 24125 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.484915 24124 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.489032 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.489771 24006 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.490212 24125 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20240426 13:17:40.491216 24044 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.491516 24125 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.492426 24132 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.492798 24132 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.493167 24132 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.506583 24125 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.521322 24132 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.525436 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.526082 24132 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
W20240426 13:17:40.526758 24045 leader_election.cc:341] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.525548 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.527477 24006 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.527968 24125 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.528424 24125 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.001s
I20240426 13:17:40.529295 24123 tablet_bootstrap.cc:492] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.529779 24123 ts_tablet_manager.cc:1367] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.064s user 0.036s sys 0.016s
I20240426 13:17:40.531131 24125 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.531858 24125 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.531881 24006 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.532295 24125 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.532512 24123 raft_consensus.cc:348] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.533264 24044 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.533526 24123 raft_consensus.cc:374] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.533998 24123 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.534035 24124 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.533149 24125 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.534740 24124 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.060s user 0.021s sys 0.015s
I20240426 13:17:40.534961 24132 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.535578 24123 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.538118 24123 ts_tablet_manager.cc:1398] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 13:17:40.539057 24123 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.544752 24123 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.545385 24125 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.017s user 0.001s sys 0.005s
I20240426 13:17:40.546375 24125 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.549896 24124 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.550650 24124 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.550935 24124 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.551659 24124 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.553805 24125 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.557114 24123 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.557605 24123 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240426 13:17:40.558167 24132 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.558425 24124 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.023s user 0.006s sys 0.003s
I20240426 13:17:40.559296 24124 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.558959 24132 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.560410 24123 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.561136 24123 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.561476 24123 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.562572 24123 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.565017 24123 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:17:40.566376 24123 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.568895 24124 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.573118 24123 tablet_bootstrap.cc:654] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.575631 24125 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.576145 24125 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.005s
I20240426 13:17:40.578909 24125 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.579644 24125 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.580008 24125 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.581367 24124 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.580786 24125 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.581786 24124 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.006s
I20240426 13:17:40.580941 23826 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.584755 24124 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.585563 24124 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.585932 24124 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.588095 24132 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.588629 24132 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.590744 24125 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20240426 13:17:40.591078 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.591629 24125 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
W20240426 13:17:40.592309 24045 leader_election.cc:341] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.593173 24132 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.586966 24124 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.595175 24124 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240426 13:17:40.595266 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.595860 24006 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.596057 24124 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.597549 24044 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.598392 24125 tablet_bootstrap.cc:654] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.598493 24142 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.598940 24142 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.599238 24142 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.604692 24123 tablet_bootstrap.cc:492] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.605132 24123 ts_tablet_manager.cc:1367] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.007s
I20240426 13:17:40.605221 24142 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.608196 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
W20240426 13:17:40.609408 24045 leader_election.cc:341] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.610217 24142 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.611091 24124 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.611184 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.611757 24006 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.617350 24123 raft_consensus.cc:348] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.618044 24123 raft_consensus.cc:374] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.618299 24123 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.618966 24123 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.619400 24006 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.621730 24123 ts_tablet_manager.cc:1398] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240426 13:17:40.625308 24123 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.625814 24044 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters: cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.627259 24142 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.627696 24125 tablet_bootstrap.cc:492] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.628218 24125 ts_tablet_manager.cc:1367] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.009s
I20240426 13:17:40.628124 24142 raft_consensus.cc:686] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.629668 24142 consensus_queue.cc:237] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.631338 24123 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.631227 24125 raft_consensus.cc:348] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.631976 24125 raft_consensus.cc:374] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.637549 24125 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.640313 24125 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.642741 24124 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.644747 24124 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.021s
I20240426 13:17:40.647562 24124 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.644309 24125 ts_tablet_manager.cc:1398] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20240426 13:17:40.648147 24123 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.649204 24124 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.649338 24123 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.005s
I20240426 13:17:40.650173 24125 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.652356 24123 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.653203 24123 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.653538 24123 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.654294 24123 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.656610 24123 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240426 13:17:40.657042 24125 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.657387 24124 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.659981 24123 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.665285 24124 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.667920 23826 catalog_manager.cc:5484] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:40.671897 24124 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20240426 13:17:40.673027 24124 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.678256 24125 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.678853 24125 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.012s
I20240426 13:17:40.679518 24124 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.679759 24142 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.680233 24142 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.673522 24123 tablet_bootstrap.cc:654] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.682191 24142 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.683271 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.684773 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
W20240426 13:17:40.686892 24045 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.691648 24123 tablet_bootstrap.cc:492] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.691725 24124 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.692204 24123 ts_tablet_manager.cc:1367] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.005s
W20240426 13:17:40.692238 24044 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.692278 24124 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240426 13:17:40.692860 24044 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996; no voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.693710 24142 raft_consensus.cc:2726] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.694716 24128 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.695091 24128 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.695683 24125 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696139 24123 raft_consensus.cc:348] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696498 24125 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.696573 24124 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.696892 24125 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.697432 24128 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.704943 24124 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.705519 24124 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.706388 24124 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.706038 24125 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.709203 24124 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20240426 13:17:40.709662 24142 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.710098 24142 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.711987 24142 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.713266 24124 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.713191 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.713949 24006 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.716476 24044 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.717259 24142 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.717613 24142 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.717948 24142 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.718115 24125 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20240426 13:17:40.719221 24125 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.696862 24123 raft_consensus.cc:374] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.722208 24123 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.723335 24123 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.724625 24142 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.726656 24142 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.728320 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.728899 24006 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.730126 24124 tablet_bootstrap.cc:654] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.726730 24123 ts_tablet_manager.cc:1398] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.034s user 0.005s sys 0.004s
I20240426 13:17:40.731741 24125 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.735697 24123 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.742617 24006 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.743793 24044 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.744581 24142 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.745050 24142 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:40.745836 24142 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.750820 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.751441 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:40.752692 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.753249 23931 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.758877 24123 tablet_bootstrap.cc:654] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.764199 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.764541 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:40.765303 24006 raft_consensus.cc:2379] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.769855 24131 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.770892 24131 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.778064 24131 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.781931 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.784618 23895 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.786084 24128 raft_consensus.cc:2726] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.785290 23828 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:40.790382 24128 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.790961 24128 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.793308 24128 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.795328 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.796005 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.796365 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.797003 24006 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.797137 23895 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:40.797964 24128 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.798327 24128 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.798671 24128 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.799355 24131 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.799945 24131 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.805765 24131 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.806064 24128 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.809316 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.809512 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.809922 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.810245 24006 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.817494 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.818588 23895 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:40.808163 24128 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.821492 24128 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.821935 24128 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:40.822804 24128 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.824688 24125 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.825289 24125 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.106s user 0.021s sys 0.010s
I20240426 13:17:40.819589 24006 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.831277 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.833611 24079 raft_consensus.cc:2379] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate decbe9ee3d0a47bab14b976ecaf12843 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:40.825771 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.833878 24125 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.834858 24080 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.834925 24125 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.835625 24125 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.836572 23970 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.836503 24125 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.837785 24131 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.838268 24131 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.841248 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.841926 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.842932 24166 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:17:40.843544 23971 leader_election.cc:341] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.838713 24131 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.843581 24166 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.844468 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:40.845965 24166 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.847358 24170 raft_consensus.cc:2726] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.847841 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.850073 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.850849 24006 raft_consensus.cc:2379] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.851436 24124 tablet_bootstrap.cc:492] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.851894 24124 ts_tablet_manager.cc:1367] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.139s user 0.032s sys 0.047s
I20240426 13:17:40.852865 24125 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.027s user 0.003s sys 0.008s
I20240426 13:17:40.853905 24125 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.854795 24124 raft_consensus.cc:348] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.855518 24124 raft_consensus.cc:374] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.855814 24124 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.856549 24124 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.859149 24124 ts_tablet_manager.cc:1398] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:17:40.860430 24125 tablet_bootstrap.cc:654] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.861549 24123 tablet_bootstrap.cc:492] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.861414 23895 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.861980 24123 ts_tablet_manager.cc:1367] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.000s
I20240426 13:17:40.862337 24167 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.867548 24123 raft_consensus.cc:348] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.868283 24123 raft_consensus.cc:374] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.868557 24123 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.869130 24123 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.870573 24167 raft_consensus.cc:2726] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.871124 24123 ts_tablet_manager.cc:1398] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240426 13:17:40.872294 24124 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.875572 24131 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.871971 24123 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Bootstrap starting.
I20240426 13:17:40.878885 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.879729 23930 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.880357 24128 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.880774 24128 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.882715 24128 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.884083 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.884675 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.885305 24131 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.885051 24123 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.886083 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.885502 24080 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.886776 24079 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.888083 23895 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:40.888926 24166 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240426 13:17:40.889070 23895 leader_election.cc:341] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.889286 24166 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.889642 24166 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.893066 23930 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.879007 23826 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.893752 24166 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.894413 23971 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.895148 24131 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.895537 24131 raft_consensus.cc:686] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:40.896327 24131 consensus_queue.cc:237] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.900121 24166 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.901618 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.904850 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.905578 24079 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.909654 24123 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: No bootstrap required, opened a new log
I20240426 13:17:40.910115 24123 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.000s
I20240426 13:17:40.912734 24123 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.913445 24123 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.913386 24131 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.913830 24123 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.914012 24124 tablet_bootstrap.cc:654] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.914125 24131 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.914639 24123 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.915489 24080 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.916134 24131 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.916909 24123 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20240426 13:17:40.920120 23895 leader_election.cc:341] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.920874 24166 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.922049 24166 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:40.919203 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:40.929186 24166 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:40.929626 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.911305 23827 catalog_manager.cc:5484] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 0 to 1, leader changed from <none> to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.932754 24078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.933616 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.937009 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.937104 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 0.
I20240426 13:17:40.937363 23971 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.937870 24006 raft_consensus.cc:2379] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cad2f0c769574fa4ba4bd895fda5a10e in current term 1: Already voted for candidate 20b204c02fe14d9eb754d5a169d43996 in this term.
I20240426 13:17:40.938138 24125 tablet_bootstrap.cc:492] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.938501 24125 ts_tablet_manager.cc:1367] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.085s user 0.035s sys 0.019s
I20240426 13:17:40.939337 24131 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.939705 24131 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.940058 24131 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.940725 24125 raft_consensus.cc:348] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.940066 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e; no voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.941354 24125 raft_consensus.cc:374] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.941659 24125 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.942044 24166 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.942232 24125 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.944191 24125 ts_tablet_manager.cc:1398] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240426 13:17:40.944962 24125 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Bootstrap starting.
I20240426 13:17:40.950417 24125 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.946703 24131 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.952248 24166 raft_consensus.cc:2726] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:40.954147 24079 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.955143 23895 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters: decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:40.956998 24124 tablet_bootstrap.cc:492] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:40.957202 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:40.957520 24124 ts_tablet_manager.cc:1367] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.024s
I20240426 13:17:40.957816 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.957676 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:40.959199 24079 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.960283 24124 raft_consensus.cc:348] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.960930 24124 raft_consensus.cc:374] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.961175 24124 raft_consensus.cc:729] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:40.956167 24131 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.961752 24124 consensus_queue.cc:260] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.964941 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.967144 23971 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.967911 24131 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.965364 24166 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:40.968281 24125 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: No bootstrap required, opened a new log
I20240426 13:17:40.968524 24128 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:40.968746 24166 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:40.968868 24125 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.006s
I20240426 13:17:40.968992 24128 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.969594 24166 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.970919 24128 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:40.972637 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:40.972734 24131 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:40.973155 24124 ts_tablet_manager.cc:1398] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.015s user 0.002s sys 0.005s
I20240426 13:17:40.972812 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:40.971328 24125 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.973938 24125 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:40.974231 24125 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Initialized, Role: FOLLOWER
W20240426 13:17:40.974380 23895 leader_election.cc:341] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:40.974623 24131 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.974895 24125 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.975782 24006 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 0.
I20240426 13:17:40.978240 24124 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Bootstrap starting.
I20240426 13:17:40.978781 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:40.979754 24167 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:40.980211 24167 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:40.980522 24167 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.983834 24124 tablet_bootstrap.cc:654] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Neither blocks nor log segments found. Creating new log.
I20240426 13:17:40.965080 24079 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 1.
I20240426 13:17:40.987303 24167 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:40.990113 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:40.990710 24006 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:40.991040 23826 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 0 to 1, leader changed from <none> to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:40.997802 24006 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 1.
I20240426 13:17:40.998854 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:40.999657 24128 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:41.007465 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
W20240426 13:17:41.008878 23895 leader_election.cc:341] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:17:41.008339 24125 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20240426 13:17:41.008410 24124 tablet_bootstrap.cc:492] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: No bootstrap required, opened a new log
I20240426 13:17:41.010556 24124 ts_tablet_manager.cc:1367] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20240426 13:17:41.008908 24167 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 1 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.013924 24124 raft_consensus.cc:348] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.019982 24124 raft_consensus.cc:374] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:17:41.014430 24131 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.020349 24124 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Initialized, Role: FOLLOWER
I20240426 13:17:41.020746 24131 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.021150 24124 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.022763 24131 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.014159 24128 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.023754 24128 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.024312 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b204c02fe14d9eb754d5a169d43996" is_pre_election: true
I20240426 13:17:41.025086 24079 raft_consensus.cc:2379] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate decbe9ee3d0a47bab14b976ecaf12843 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:41.025153 24142 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.028018 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.030153 23971 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: decbe9ee3d0a47bab14b976ecaf12843; no voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.031363 24131 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.032721 24124 ts_tablet_manager.cc:1398] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: Time spent starting tablet: real 0.022s user 0.008s sys 0.003s
I20240426 13:17:41.032577 23827 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.037680 24131 raft_consensus.cc:2726] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:41.056020 24142 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.056169 23826 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 0 to 1, leader changed from <none> to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.073875 24142 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.087780 24132 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.147023 24187 raft_consensus.cc:482] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.147588 24187 raft_consensus.cc:504] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.150357 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.150527 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:41.150959 24187 leader_election.cc:288] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.151036 23930 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:41.151132 24006 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 0.
I20240426 13:17:41.152173 24045 leader_election.cc:302] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:41.152927 24187 raft_consensus.cc:2781] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:17:41.153264 24187 raft_consensus.cc:482] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:17:41.153584 24187 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.159864 24187 raft_consensus.cc:504] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.161731 24187 leader_election.cc:288] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.162462 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.162722 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ada48e64e5a4d36b5186827ace67fb9" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.162922 23930 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.163324 24006 raft_consensus.cc:3037] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.166985 23930 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:41.167841 24045 leader_election.cc:302] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:41.168490 24187 raft_consensus.cc:2781] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:17:41.168870 24187 raft_consensus.cc:686] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.169364 24006 raft_consensus.cc:2454] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 1.
I20240426 13:17:41.169711 24187 consensus_queue.cc:237] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.177804 23827 catalog_manager.cc:5484] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:41.201128 24187 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:17:41.201599 24187 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.203087 24187 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.203963 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" is_pre_election: true
I20240426 13:17:41.204090 24006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843" is_pre_election: true
I20240426 13:17:41.204876 24006 raft_consensus.cc:2379] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20b204c02fe14d9eb754d5a169d43996 in current term 1: Already voted for candidate cad2f0c769574fa4ba4bd895fda5a10e in this term.
I20240426 13:17:41.205694 24044 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996; no voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.206307 24187 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:17:41.212432 24187 raft_consensus.cc:2726] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:17:41.322573 24131 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.323727 24187 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.341476 24187 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.341697 24170 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.346462 24166 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.370002 24179 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.370404 24170 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.382313 24131 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.420261 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.420964 24079 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.428406 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.428953 24079 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.441273 24006 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:43458
I20240426 13:17:41.441722 24006 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:41.453980 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:41.454629 23930 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.458812 24079 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:41.459328 24079 raft_consensus.cc:595] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.465900 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:41.466481 23930 raft_consensus.cc:595] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:41.485009 24187 raft_consensus.cc:979] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.485445 24140 raft_consensus.cc:1067] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.486796 24006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:41.487227 24006 raft_consensus.cc:482] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.487534 24006 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.491685 24006 raft_consensus.cc:504] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.493429 24006 leader_election.cc:288] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.493885 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.494012 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e6fe9ec33c40f8aa98d61394752d08" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.494437 23930 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.494590 24079 raft_consensus.cc:3032] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.494939 24079 raft_consensus.cc:729] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.495644 24079 consensus_queue.cc:260] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.497144 24079 raft_consensus.cc:3037] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.498561 23930 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.499471 23971 leader_election.cc:302] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:41.500082 24131 raft_consensus.cc:2781] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.500453 24131 raft_consensus.cc:686] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.501082 24131 consensus_queue.cc:237] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.503502 24079 raft_consensus.cc:2454] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.507874 23825 catalog_manager.cc:5484] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.521039 24166 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.531581 24166 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.532006 24128 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.547852 24140 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.548314 24140 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.553309 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34980
I20240426 13:17:41.553867 23930 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.554216 23930 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.566092 23930 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.567726 24166 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:17:41.568600 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.569327 24079 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.569542 23930 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:41.569710 24079 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:41.570920 24079 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.572525 24079 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.573963 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.574579 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.579485 24079 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.581022 24005 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.580852 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:41.582432 24166 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.582906 24166 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.583678 24166 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:41.589185 24166 raft_consensus.cc:979] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.589602 24167 raft_consensus.cc:1067] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.591127 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:41.591643 24005 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.591984 24005 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.591851 23825 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.598505 24005 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.601707 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:41.601914 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.602298 23930 raft_consensus.cc:3032] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.602423 24005 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.602632 23930 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:41.602725 24079 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.603399 23930 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.604626 23930 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.609084 24079 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.610209 23970 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:41.610921 23930 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:41.611100 24131 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:41.611627 24131 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.612394 24131 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.617591 24142 consensus_queue.cc:1035] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.619428 23825 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:41.631590 24187 consensus_queue.cc:1035] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:17:41.817775 24131 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.818275 24169 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.819852 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:41.820490 23930 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.820853 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.887712 23930 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.889559 23930 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.890774 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:41.891290 24005 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:41.891608 24005 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:41.891965 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:41.892534 24079 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.892396 24005 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.893626 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.952190 24142 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.952637 24187 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:41.954231 23930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34980
I20240426 13:17:41.954689 23930 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:41.954975 23930 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:41.992882 24004 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:41.993358 24005 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.994407 23930 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:41.995290 24128 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:41.996317 23930 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:41.996408 24166 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.996727 24079 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:41.996996 24166 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:41.997900 23895 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:41.998672 24166 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.000012 24166 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.000352 24004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:42.000954 24004 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:42.001327 24004 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.004118 24079 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.004739 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:42.005331 24080 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:42.005681 24080 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:42.006449 24080 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.001279 24166 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.007676 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.006831 24169 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:17:42.009472 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:42.010073 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.014720 23930 raft_consensus.cc:1261] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.016172 24079 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.017751 24179 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.018899 24170 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.024075 23931 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.026100 24169 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.029770 23825 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.034168 24079 raft_consensus.cc:1261] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.035902 24169 consensus_queue.cc:1035] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.093554 24080 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:42.093595 24004 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.098104 23895 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:42.098181 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:42.099027 23931 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:42.099172 24166 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.099416 23931 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.099730 24166 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:42.099776 24004 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:42.100734 24166 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.101576 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:42.100250 23931 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.102166 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.102843 23931 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:42.107352 23825 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 1 to 2, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.192867 24005 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 2.
I20240426 13:17:42.193125 23931 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:42.192871 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 2.
I20240426 13:17:42.194376 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:42.195351 24169 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:42.195837 24169 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:42.196569 24169 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:42.202766 23825 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:42.411480 24005 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.412662 24179 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.420836 24080 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.424335 24166 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.663064 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.664139 24170 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.671391 24080 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.673247 24170 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.682006 24080 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.683010 24166 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:42.692304 24005 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:42.693842 24179 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:44.490451 23796 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240426 13:17:44.491189 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:44.510504 23931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:50698
I20240426 13:17:44.510989 23931 raft_consensus.cc:595] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:44.520948 24004 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:44.521549 24004 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:44.522707 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 13:17:44.523098 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:44.645644 24169 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:44.646091 24215 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:44.647428 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:44.647872 24080 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:44.648181 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.652292 24080 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.653923 24080 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:44.654680 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:44.655253 23931 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.655982 24004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:44.656558 24004 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:44.656889 24004 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:44.657589 24004 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.658797 24004 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:44.661572 23931 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 3.
I20240426 13:17:44.662607 24045 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:44.663331 24140 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:44.664542 24004 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 3.
I20240426 13:17:44.664516 24140 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:44.665427 24140 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:44.671753 23828 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 2 to 3, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:45.008005 24179 raft_consensus.cc:979] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:45.008391 24225 raft_consensus.cc:1067] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:45.009616 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:44396
I20240426 13:17:45.010028 24080 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:45.010368 24080 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.014817 24080 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.016273 24080 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:45.016971 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:45.017172 24004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:45.017434 23931 raft_consensus.cc:3032] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:45.017670 24004 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.017730 23931 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:45.018440 23931 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.019361 23931 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:45.022750 24004 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:45.023609 24044 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:45.024402 24140 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:45.024796 23931 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:45.024853 24140 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:45.025806 24140 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:45.034741 23828 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 1 to 2, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:45.123018 24004 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:45.124135 24140 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:45.131906 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:45.133278 24219 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:45.523866 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:45.524454 24004 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:45.525841 24140 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:45.531371 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:45.531926 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:45.540438 23931 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:45.541671 24140 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:46.532848 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:46.538168 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:46.538589 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:47.539420 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:47.544373 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:47.544760 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:48.545725 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:48.551060 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:48.551445 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:49.552165 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:49.556962 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:49.557371 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:50.558069 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:50.564134 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:50.564656 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:51.565469 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:51.570698 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:51.571151 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:52.571970 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:52.576527 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:52.576835 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:53.577527 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:53.582331 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:53.582777 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:54.602828 23796 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240426 13:17:54.606851 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.607470 23930 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.611738 24080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:54.612191 24080 raft_consensus.cc:595] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.615906 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.616338 23930 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.619841 23930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:34988
I20240426 13:17:54.620218 23930 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.623816 24080 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
mode: GRACEFUL
new_leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:44420
I20240426 13:17:54.624223 24080 raft_consensus.cc:595] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Received request to transfer leadership to TS decbe9ee3d0a47bab14b976ecaf12843
I20240426 13:17:54.653193 24246 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.653652 24248 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.654886 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.655336 24005 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.655706 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.661381 24005 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.662964 24005 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.663605 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.663826 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.664546 24080 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.664501 23930 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.665051 23930 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.665647 23930 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.666494 23930 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.668794 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.669734 23970 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:54.670396 24215 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.670598 23930 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.671300 24215 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.672159 24215 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.678185 23825 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.715777 24140 raft_consensus.cc:979] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.716389 24256 raft_consensus.cc:1067] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.717639 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:54.718068 24005 raft_consensus.cc:482] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.718322 24005 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.722512 24005 raft_consensus.cc:504] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.723902 24005 leader_election.cc:288] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.724635 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.724731 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc471e293e547cca28b0aecdccbb30d" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.725143 23930 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.725210 24080 raft_consensus.cc:3032] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.725571 24080 raft_consensus.cc:729] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:54.726256 24080 consensus_queue.cc:260] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.727332 24080 raft_consensus.cc:3037] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.730640 23930 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.731540 23971 leader_election.cc:302] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:54.732156 24215 raft_consensus.cc:2781] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.732348 24080 raft_consensus.cc:2454] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.732578 24215 raft_consensus.cc:686] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.733283 24215 consensus_queue.cc:237] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.739053 23825 catalog_manager.cc:5484] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.771231 24246 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.771636 24248 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.772786 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.773219 24005 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.773486 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.777645 24005 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.778982 24005 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.779702 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.779758 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.780265 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.780370 23930 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.780675 23930 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.781149 23930 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.781966 23930 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.784163 24080 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.784881 23970 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:54.785588 24215 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.785912 24215 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.786190 23930 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.786605 24215 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.792882 23825 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:54.890141 24246 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.890628 24248 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.891773 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43454
I20240426 13:17:54.892216 24005 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.892454 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.896148 24005 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.897382 24005 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541)
I20240426 13:17:54.898182 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.898360 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.898715 24080 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.898830 23930 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:54.899152 23930 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:54.899818 23930 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.900913 23930 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:54.904460 24080 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.905472 23970 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:54.906147 24215 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:54.906276 23930 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 3.
I20240426 13:17:54.906589 24215 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.907301 24215 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:54.913605 23825 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 2 to 3, leader changed from cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:54.966085 24255 raft_consensus.cc:979] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: : Instructing follower decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.966459 24256 raft_consensus.cc:1067] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Signalling peer decbe9ee3d0a47bab14b976ecaf12843 to start an election
I20240426 13:17:54.967638 24005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f60b45962d422292f2f367e49d0376"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
from {username='slave'} at 127.0.0.1:43452
I20240426 13:17:54.968047 24005 raft_consensus.cc:482] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:54.968286 24005 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.972147 24005 raft_consensus.cc:504] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.973471 24005 leader_election.cc:288] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 4 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:54.974351 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:54.974412 23930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f60b45962d422292f2f367e49d0376" candidate_uuid: "decbe9ee3d0a47bab14b976ecaf12843" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:54.974912 24080 raft_consensus.cc:3032] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:54.974941 23930 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.975261 24080 raft_consensus.cc:729] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:54.975876 24080 consensus_queue.cc:260] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.976727 24080 raft_consensus.cc:3037] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:54.978956 23930 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 4.
I20240426 13:17:54.979806 23971 leader_election.cc:302] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:54.980526 24215 raft_consensus.cc:2781] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:54.980854 24215 raft_consensus.cc:686] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 LEADER]: Becoming Leader. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:54.981485 24080 raft_consensus.cc:2454] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate decbe9ee3d0a47bab14b976ecaf12843 in term 4.
I20240426 13:17:54.981526 24215 consensus_queue.cc:237] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:54.987355 23825 catalog_manager.cc:5484] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 reported cstate change: term changed from 3 to 4, leader changed from 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3) to decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "decbe9ee3d0a47bab14b976ecaf12843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:55.156229 23930 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.157480 24249 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.166864 24080 raft_consensus.cc:1261] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.168776 24249 consensus_queue.cc:1035] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.238744 24080 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.239775 24249 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.247682 23930 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.249034 24249 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.259029 23930 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.260160 24249 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.268187 24080 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.269809 24249 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:17:55.330651 24080 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.331707 24249 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.341310 23930 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:55.342682 24272 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:55.445375 23931 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:55.446470 24272 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:55.453888 24080 raft_consensus.cc:1261] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer decbe9ee3d0a47bab14b976ecaf12843: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:55.455201 24272 consensus_queue.cc:1035] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:57.651772 23796 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240426 13:17:57.652379 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:57.658703 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.659121 24005 raft_consensus.cc:595] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:57.661029 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.661392 24005 raft_consensus.cc:595] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.663005 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.663352 24005 raft_consensus.cc:595] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Received request to transfer leadership to TS 20b204c02fe14d9eb754d5a169d43996
I20240426 13:17:57.665069 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.665438 24005 raft_consensus.cc:595] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.666989 24005 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
mode: GRACEFUL
new_leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:58552
I20240426 13:17:57.667336 24005 raft_consensus.cc:595] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Received request to transfer leadership to TS cad2f0c769574fa4ba4bd895fda5a10e
I20240426 13:17:57.668148 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:17:57.668519 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:57.823755 24277 raft_consensus.cc:979] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:57.824172 24215 raft_consensus.cc:1067] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:57.825237 24080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:57.825642 24080 raft_consensus.cc:482] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:57.825877 24080 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.829583 24080 raft_consensus.cc:504] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.831228 24080 leader_election.cc:288] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 4 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:57.831979 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:57.832021 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52fc238a1b814a388b781f80b802ba71" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:57.832604 24005 raft_consensus.cc:3032] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:57.832901 24005 raft_consensus.cc:729] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:57.833248 23931 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.833415 24005 consensus_queue.cc:260] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.834266 24005 raft_consensus.cc:3037] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.837296 23931 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 4.
I20240426 13:17:57.837980 24005 raft_consensus.cc:2454] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 4.
I20240426 13:17:57.838317 24045 leader_election.cc:302] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:57.839063 24256 raft_consensus.cc:2781] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:57.839959 24256 raft_consensus.cc:686] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:57.840703 24256 consensus_queue.cc:237] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:57.846005 23828 catalog_manager.cc:5484] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:57.970664 24285 raft_consensus.cc:979] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:57.971367 24215 raft_consensus.cc:1067] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:57.972764 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:57.973305 23931 raft_consensus.cc:482] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:57.973623 23931 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.978976 23931 raft_consensus.cc:504] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:57.981030 23931 leader_election.cc:288] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Requested vote from peers 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:57.982265 24079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:57.982609 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9fc30fbb8de42d58ac588a5f723f4fe" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:57.983199 24005 raft_consensus.cc:3032] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:57.983253 24079 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.983597 24005 raft_consensus.cc:729] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:57.984512 24005 consensus_queue.cc:260] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:57.985702 24005 raft_consensus.cc:3037] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:57.990443 24079 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:57.991253 24080 raft_consensus.cc:1226] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Rejecting Update request from peer decbe9ee3d0a47bab14b976ecaf12843 for earlier term 3. Current term is 4. Ops: []
I20240426 13:17:57.991881 23895 leader_election.cc:302] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:57.992527 24005 raft_consensus.cc:2454] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:57.994555 24286 raft_consensus.cc:2781] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:57.995898 24286 raft_consensus.cc:686] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:57.997088 24286 consensus_queue.cc:237] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } }
I20240426 13:17:58.006286 23828 catalog_manager.cc:5484] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.063885 24285 raft_consensus.cc:979] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.064328 24215 raft_consensus.cc:1067] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.065466 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:58.065855 23931 raft_consensus.cc:482] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.066088 23931 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.069998 23931 raft_consensus.cc:504] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.071161 23931 leader_election.cc:288] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:58.071873 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.072144 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b6922ff368484e93a5442573ad8b85" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:58.072353 24005 raft_consensus.cc:3032] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:17:58.072633 24005 raft_consensus.cc:729] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.072652 24080 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.073293 24005 consensus_queue.cc:260] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.074258 24005 raft_consensus.cc:3037] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:17:58.078260 24005 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:58.078986 23895 leader_election.cc:302] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cad2f0c769574fa4ba4bd895fda5a10e, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:58.079006 24080 raft_consensus.cc:2454] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 4.
I20240426 13:17:58.079671 24286 raft_consensus.cc:2781] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:17:58.080137 24286 raft_consensus.cc:686] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:58.080776 24286 consensus_queue.cc:237] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.086335 23828 catalog_manager.cc:5484] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 3 to 4, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.100432 24285 raft_consensus.cc:979] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.100759 24215 raft_consensus.cc:1067] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Signalling peer cad2f0c769574fa4ba4bd895fda5a10e to start an election
I20240426 13:17:58.101894 23931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71"
dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
from {username='slave'} at 127.0.0.1:34986
I20240426 13:17:58.102252 23931 raft_consensus.cc:482] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.102471 23931 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.106499 23931 raft_consensus.cc:504] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.107430 24285 raft_consensus.cc:979] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: : Instructing follower 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:58.107889 24215 raft_consensus.cc:1067] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Signalling peer 20b204c02fe14d9eb754d5a169d43996 to start an election
I20240426 13:17:58.107957 23931 leader_election.cc:288] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 3 election: Requested vote from peers decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101), 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3:45215)
I20240426 13:17:58.108556 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.108862 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebc1d91fea64133989f26f880fdec71" candidate_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
I20240426 13:17:58.109091 24005 raft_consensus.cc:3032] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:17:58.109458 24005 raft_consensus.cc:729] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.109405 24080 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.109376 24079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e"
dest_uuid: "20b204c02fe14d9eb754d5a169d43996"
from {username='slave'} at 127.0.0.1:44404
I20240426 13:17:58.110347 24079 raft_consensus.cc:482] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:17:58.110792 24079 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.110612 24005 consensus_queue.cc:260] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.111886 24005 raft_consensus.cc:3037] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:17:58.115483 24080 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 3.
I20240426 13:17:58.116228 23895 leader_election.cc:302] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, cad2f0c769574fa4ba4bd895fda5a10e; no voters:
I20240426 13:17:58.116760 24286 raft_consensus.cc:2781] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:17:58.117153 24286 raft_consensus.cc:686] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 LEADER]: Becoming Leader. State: Replica: cad2f0c769574fa4ba4bd895fda5a10e, State: Running, Role: LEADER
I20240426 13:17:58.117688 24005 raft_consensus.cc:2454] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad2f0c769574fa4ba4bd895fda5a10e in term 3.
I20240426 13:17:58.116859 24079 raft_consensus.cc:504] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.117902 24286 consensus_queue.cc:237] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.119668 24079 leader_election.cc:288] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Requested vote from peers cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541), decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101)
I20240426 13:17:58.119921 23931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cad2f0c769574fa4ba4bd895fda5a10e"
I20240426 13:17:58.120193 24005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734eaab0271041ecaa0e13f446948d2e" candidate_uuid: "20b204c02fe14d9eb754d5a169d43996" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "decbe9ee3d0a47bab14b976ecaf12843"
I20240426 13:17:58.120644 23931 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.120995 24005 raft_consensus.cc:3032] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:17:58.121238 24005 raft_consensus.cc:729] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: decbe9ee3d0a47bab14b976ecaf12843, State: Running, Role: LEADER
I20240426 13:17:58.121755 24005 consensus_queue.cc:260] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.122679 24005 raft_consensus.cc:3037] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:17:58.126248 24005 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:58.127202 24044 leader_election.cc:302] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b204c02fe14d9eb754d5a169d43996, decbe9ee3d0a47bab14b976ecaf12843; no voters:
I20240426 13:17:58.127858 23931 raft_consensus.cc:2454] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b204c02fe14d9eb754d5a169d43996 in term 2.
I20240426 13:17:58.131011 24256 raft_consensus.cc:2781] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:17:58.131314 24256 raft_consensus.cc:686] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Becoming Leader. State: Replica: 20b204c02fe14d9eb754d5a169d43996, State: Running, Role: LEADER
I20240426 13:17:58.132089 24256 consensus_queue.cc:237] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } }
I20240426 13:17:58.133785 23828 catalog_manager.cc:5484] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e reported cstate change: term changed from 2 to 3, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1). New cstate: current_term: 3 leader_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20240426 13:17:58.139058 23825 catalog_manager.cc:5484] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 reported cstate change: term changed from 1 to 2, leader changed from decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2) to 20b204c02fe14d9eb754d5a169d43996 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "20b204c02fe14d9eb754d5a169d43996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 } health_report { overall_health: HEALTHY } } }
I20240426 13:17:58.404616 24079 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.406026 24286 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.415146 24005 raft_consensus.cc:1261] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.416540 24286 consensus_queue.cc:1035] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.431103 24005 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.432204 24256 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.440541 23931 raft_consensus.cc:1261] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.441776 24256 consensus_queue.cc:1035] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.489435 24079 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.490466 24286 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.498703 24005 raft_consensus.cc:1261] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:17:58.499966 24286 consensus_queue.cc:1035] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:17:58.573761 24079 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:58.574909 24301 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b204c02fe14d9eb754d5a169d43996" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:58.580253 23931 raft_consensus.cc:1261] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:58.583189 24256 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad2f0c769574fa4ba4bd895fda5a10e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:17:58.584244 24005 raft_consensus.cc:1261] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Refusing update from remote peer cad2f0c769574fa4ba4bd895fda5a10e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:17:58.587536 24286 consensus_queue.cc:1035] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:17:58.590664 24004 raft_consensus.cc:1261] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Refusing update from remote peer 20b204c02fe14d9eb754d5a169d43996: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:17:58.593616 24278 consensus_queue.cc:1035] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "decbe9ee3d0a47bab14b976ecaf12843" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:17:58.669427 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:58.674559 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:58.674939 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:17:59.675603 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:17:59.680959 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:17:59.681352 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:00.682124 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:00.686605 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:00.686975 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:01.687716 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:01.692402 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:01.692787 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:02.775274 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:02.780146 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:02.780464 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 13:18:03.358970 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800171060 after lost signal to thread 23799
W20240426 13:18:03.359694 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800171740 after lost signal to thread 23862
W20240426 13:18:03.360456 24101 debug-util.cc:398] Leaking SignalData structure 0x7b0800080960 after lost signal to thread 23955
W20240426 13:18:03.361138 24101 debug-util.cc:398] Leaking SignalData structure 0x7b08001f25a0 after lost signal to thread 24030
W20240426 13:18:03.361752 24101 debug-util.cc:398] Leaking SignalData structure 0x7b080011efc0 after lost signal to thread 24104
I20240426 13:18:03.781195 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:03.786418 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:03.786772 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:04.787554 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:04.792912 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:04.793283 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:05.794121 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:05.798740 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:05.799149 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:06.799944 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:06.804607 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:06.805001 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:07.824869 23796 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240426 13:18:07.825474 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
I20240426 13:18:07.847337 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:07.848044 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.848973 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.849563 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.852523 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.853229 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.853577 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.854979 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.855384 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.855978 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.857324 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.857712 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.858039 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P cad2f0c769574fa4ba4bd895fda5a10e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.859324 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.859716 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.860297 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.861517 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.861937 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:07.862495 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.863826 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.864312 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.864672 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P cad2f0c769574fa4ba4bd895fda5a10e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.866003 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.866372 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.866711 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P cad2f0c769574fa4ba4bd895fda5a10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:07.868052 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e: stopping tablet replica
I20240426 13:18:07.868476 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:07.868826 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P cad2f0c769574fa4ba4bd895fda5a10e [term 3 FOLLOWER]: Raft consensus is shut down!
W20240426 13:18:07.975165 24045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111)
W20240426 13:18:07.978703 24045 consensus_peers.cc:485] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:07.979096 24045 consensus_peers.cc:485] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:07.985519 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
W20240426 13:18:08.004683 24045 consensus_peers.cc:485] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 -> Peer cad2f0c769574fa4ba4bd895fda5a10e (127.23.61.1:44541): Couldn't send request to peer cad2f0c769574fa4ba4bd895fda5a10e. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:08.008353 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
I20240426 13:18:08.033958 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:08.034828 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.035974 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.037333 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.040086 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.040917 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.041491 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P decbe9ee3d0a47bab14b976ecaf12843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.043419 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.043965 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.044473 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P decbe9ee3d0a47bab14b976ecaf12843 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.046082 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.046487 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.046936 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.048570 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.049010 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.049423 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.050899 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.051326 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.052361 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.053972 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.054364 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.055115 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P decbe9ee3d0a47bab14b976ecaf12843 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.056811 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.057260 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.057682 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.059291 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843: stopping tablet replica
I20240426 13:18:08.059671 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.060104 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P decbe9ee3d0a47bab14b976ecaf12843 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.092536 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:18:08.110172 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
W20240426 13:18:08.135170 24044 consensus_peers.cc:485] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 -> Peer decbe9ee3d0a47bab14b976ecaf12843 (127.23.61.2:35101): Couldn't send request to peer decbe9ee3d0a47bab14b976ecaf12843. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.2:35101: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:08.137109 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:08.137778 23796 tablet_replica.cc:330] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.138432 23796 raft_consensus.cc:2229] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.139289 23796 raft_consensus.cc:2258] T 6ada48e64e5a4d36b5186827ace67fb9 P 20b204c02fe14d9eb754d5a169d43996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.141184 23796 tablet_replica.cc:330] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.141639 23796 raft_consensus.cc:2229] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.142058 23796 raft_consensus.cc:2258] T 3fc471e293e547cca28b0aecdccbb30d P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.143770 23796 tablet_replica.cc:330] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.144306 23796 raft_consensus.cc:2229] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.144766 23796 raft_consensus.cc:2258] T 90f60b45962d422292f2f367e49d0376 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.146273 23796 tablet_replica.cc:330] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.146689 23796 raft_consensus.cc:2229] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.147076 23796 raft_consensus.cc:2258] T 27e6fe9ec33c40f8aa98d61394752d08 P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.148838 23796 tablet_replica.cc:330] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.149240 23796 raft_consensus.cc:2229] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.149986 23796 raft_consensus.cc:2258] T 52fc238a1b814a388b781f80b802ba71 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.151510 23796 tablet_replica.cc:330] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.151969 23796 raft_consensus.cc:2229] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.152464 23796 raft_consensus.cc:2258] T eebc1d91fea64133989f26f880fdec71 P 20b204c02fe14d9eb754d5a169d43996 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.153914 23796 tablet_replica.cc:330] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.154327 23796 raft_consensus.cc:2229] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.155058 23796 raft_consensus.cc:2258] T 734eaab0271041ecaa0e13f446948d2e P 20b204c02fe14d9eb754d5a169d43996 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.156468 23796 tablet_replica.cc:330] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.156852 23796 raft_consensus.cc:2229] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.157291 23796 raft_consensus.cc:2258] T 53b6922ff368484e93a5442573ad8b85 P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.158643 23796 tablet_replica.cc:330] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996: stopping tablet replica
I20240426 13:18:08.159008 23796 raft_consensus.cc:2229] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:08.159422 23796 raft_consensus.cc:2258] T c9fc30fbb8de42d58ac588a5f723f4fe P 20b204c02fe14d9eb754d5a169d43996 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.201002 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
I20240426 13:18:08.217303 23796 master.cc:537] Master@127.23.61.62:39733 shutting down...
I20240426 13:18:08.329535 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:08.330340 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:08.330744 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5631be9661c5413097ecac9e9e7835e5: stopping tablet replica
I20240426 13:18:08.349329 23796 master.cc:552] Master@127.23.61.62:39733 shutdown complete.
20b204c02fe14d9eb754d5a169d43996 1
cad2f0c769574fa4ba4bd895fda5a10e 4
decbe9ee3d0a47bab14b976ecaf12843 4
20b204c02fe14d9eb754d5a169d43996 3
cad2f0c769574fa4ba4bd895fda5a10e 3
decbe9ee3d0a47bab14b976ecaf12843 3
20b204c02fe14d9eb754d5a169d43996 1
cad2f0c769574fa4ba4bd895fda5a10e 0
decbe9ee3d0a47bab14b976ecaf12843 8
20b204c02fe14d9eb754d5a169d43996 3
cad2f0c769574fa4ba4bd895fda5a10e 3
decbe9ee3d0a47bab14b976ecaf12843 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30556 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240426 13:18:08.389961 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:42989
I20240426 13:18:08.391049 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.396344 24319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.397265 24320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.398036 24322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.398334 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:08.399277 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.399464 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.399610 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488399593 us; error 0 us; skew 500 ppm
I20240426 13:18:08.400179 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.402446 23796 webserver.cc:460] Webserver started at http://127.23.61.62:46033/ using document root <none> and password file <none>
I20240426 13:18:08.402936 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.403110 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.403393 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.404628 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "947ecbe50c614eaebc7282c4f49c3f4d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.408772 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240426 13:18:08.411734 24327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.412432 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240426 13:18:08.412698 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "947ecbe50c614eaebc7282c4f49c3f4d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.412943 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.445961 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:08.447157 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.482138 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:42989
I20240426 13:18:08.482219 24378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:42989 every 8 connection(s)
I20240426 13:18:08.485821 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.494596 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Bootstrap starting.
I20240426 13:18:08.498656 24379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:08.502111 24379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: No bootstrap required, opened a new log
I20240426 13:18:08.503991 24379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.504456 24379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:08.504645 24379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947ecbe50c614eaebc7282c4f49c3f4d, State: Initialized, Role: FOLLOWER
I20240426 13:18:08.505144 24379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.505559 24379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:18:08.505746 24379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:18:08.505973 24379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:08.509255 24379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.509660 24379 leader_election.cc:302] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947ecbe50c614eaebc7282c4f49c3f4d; no voters:
I20240426 13:18:08.510591 24379 leader_election.cc:288] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 13:18:08.510933 24382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:08.512231 24382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 LEADER]: Becoming Leader. State: Replica: 947ecbe50c614eaebc7282c4f49c3f4d, State: Running, Role: LEADER
I20240426 13:18:08.512804 24382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER }
I20240426 13:18:08.513396 24379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:18:08.518684 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER } }
I20240426 13:18:08.519203 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:08.523356 24384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947ecbe50c614eaebc7282c4f49c3f4d. Latest consensus state: current_term: 1 leader_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947ecbe50c614eaebc7282c4f49c3f4d" member_type: VOTER } }
I20240426 13:18:08.523761 24384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:08.525622 24391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:18:08.530272 24391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:18:08.531220 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:18:08.537955 24391 catalog_manager.cc:1347] Generated new cluster ID: e6bf7048ebc44e95b24466848876fdb7
I20240426 13:18:08.538153 24391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:18:08.557147 24391 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:18:08.558137 24391 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:18:08.569741 24391 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: Generated new TSK 0
I20240426 13:18:08.570142 24391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:18:08.597218 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.602582 24400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.603715 24401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.605633 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:08.605944 24403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.606760 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.606938 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.607082 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488607065 us; error 0 us; skew 500 ppm
I20240426 13:18:08.607604 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.609886 23796 webserver.cc:460] Webserver started at http://127.23.61.1:45717/ using document root <none> and password file <none>
I20240426 13:18:08.610329 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.610491 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.610718 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.611752 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "615118e943cc40268286df02ea3ee519"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.615875 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240426 13:18:08.618862 24408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.619557 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 13:18:08.619814 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "615118e943cc40268286df02ea3ee519"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.620096 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.638633 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:08.639763 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.641170 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.643200 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.643380 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.643582 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.643766 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.680308 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:44527
I20240426 13:18:08.680409 24470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:44527 every 8 connection(s)
I20240426 13:18:08.684490 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.691344 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.692595 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.695143 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.696328 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:08.697463 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.697715 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.697894 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488697876 us; error 0 us; skew 500 ppm
I20240426 13:18:08.698506 24471 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.698513 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.698858 24471 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.699646 24471 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.701520 23796 webserver.cc:460] Webserver started at http://127.23.61.2:46119/ using document root <none> and password file <none>
I20240426 13:18:08.701686 24344 ts_manager.cc:194] Registered new tserver with Master: 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:08.702032 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.702239 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.702474 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.703320 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40754
I20240426 13:18:08.703505 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "9d0a6220c3eb4062b0365da45223700d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.707791 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240426 13:18:08.710731 24483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.711364 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 13:18:08.711607 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "9d0a6220c3eb4062b0365da45223700d"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.711864 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.739805 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:08.740957 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.742256 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.744328 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.744524 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.744755 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.744902 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.780867 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:33815
I20240426 13:18:08.780954 24545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:33815 every 8 connection(s)
I20240426 13:18:08.785193 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:08.791112 24549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:08.793751 24550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.794888 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:08.794832 24552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:08.796121 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:08.796319 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:08.796478 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137488796461 us; error 0 us; skew 500 ppm
I20240426 13:18:08.796640 24546 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.796969 24546 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.797053 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:08.797730 24546 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.799636 24344 ts_manager.cc:194] Registered new tserver with Master: 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:08.799732 23796 webserver.cc:460] Webserver started at http://127.23.61.3:40097/ using document root <none> and password file <none>
I20240426 13:18:08.800412 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:08.800650 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:08.800956 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:08.801388 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40756
I20240426 13:18:08.802350 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.806569 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240426 13:18:08.809448 24557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.810123 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240426 13:18:08.810362 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
format_stamp: "Formatted at 2024-04-26 13:18:08 on dist-test-slave-jx77"
I20240426 13:18:08.810597 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:08.820912 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:08.821943 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:08.823215 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:08.825238 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:08.825408 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.825623 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:08.825764 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:08.861052 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:45341
I20240426 13:18:08.861146 24619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:45341 every 8 connection(s)
I20240426 13:18:08.873054 24620 heartbeater.cc:347] Connected to a master server at 127.23.61.62:42989
I20240426 13:18:08.873409 24620 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:08.874050 24620 heartbeater.cc:511] Master 127.23.61.62:42989 requested a full tablet report, sending...
I20240426 13:18:08.875710 24344 ts_manager.cc:194] Registered new tserver with Master: f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:08.876098 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012216981s
I20240426 13:18:08.877285 24344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40762
I20240426 13:18:08.879009 23796 test_util.cc:268] Using random seed: -1138823794
I20240426 13:18:08.898370 24344 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40772:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 13:18:08.901078 24344 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:18:08.966135 24585 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:08.967533 24585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.969211 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:08.970573 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.973173 24583 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:08.974462 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.977223 24584 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:08.978590 24584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.981333 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:08.981545 24579 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:08.982607 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.984297 24578 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:08.985539 24578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.989166 24580 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:08.990389 24580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.994074 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:08.996295 24579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.996454 24577 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:08.997690 24577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:08.998836 24511 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:09.000070 24511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.000183 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.003067 24505 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:09.003293 24508 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:09.004324 24505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.005411 24506 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:09.006598 24506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.009110 24507 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:09.010332 24507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.001219 24510 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:09.013177 24509 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:09.014346 24508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.014987 24509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.015859 24510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.017534 24503 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:09.018829 24503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.021173 24504 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:09.022414 24504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.028445 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:09.029744 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.042285 24635 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.074357 24436 tablet_service.cc:1467] Processing CreateTablet for tablet 445fcfc9676148b99b04c11e1724fe62 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240426 13:18:09.075729 24436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445fcfc9676148b99b04c11e1724fe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.077368 24435 tablet_service.cc:1467] Processing CreateTablet for tablet 27eaa94161a44b1ab098df62ef5f5398 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240426 13:18:09.078662 24435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27eaa94161a44b1ab098df62ef5f5398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.080965 24635 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.083711 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 9350efe2144740e7bb3f2380a2a3ed85 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240426 13:18:09.085079 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350efe2144740e7bb3f2380a2a3ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.089630 24433 tablet_service.cc:1467] Processing CreateTablet for tablet 962e435412bd4d65b99b276341083d43 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240426 13:18:09.091351 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962e435412bd4d65b99b276341083d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.097406 24429 tablet_service.cc:1467] Processing CreateTablet for tablet e4ef08e079494079bb32b9555a38e8a6 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240426 13:18:09.098904 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ef08e079494079bb32b9555a38e8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.106751 24432 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc669f73c8d44f49ddd7df04d27bce7 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240426 13:18:09.107314 24431 tablet_service.cc:1467] Processing CreateTablet for tablet 14f0e49eace343cfaef009c9f693fa3b (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240426 13:18:09.107506 24430 tablet_service.cc:1467] Processing CreateTablet for tablet cbf8550c110746688d41e86305233931 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240426 13:18:09.112295 24427 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdcbd8d33f24437ba5d07c57e09680f (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240426 13:18:09.109544 24428 tablet_service.cc:1467] Processing CreateTablet for tablet e63ce9769fcc4689babbf3d3f9de3d57 (DEFAULT_TABLE table=test-workload [id=6cbf932c7b42468ba6b846df92a08a12]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240426 13:18:09.113883 24635 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.114344 24635 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.013s
I20240426 13:18:09.114921 24432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc669f73c8d44f49ddd7df04d27bce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.115674 24431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f0e49eace343cfaef009c9f693fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.116382 24427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdcbd8d33f24437ba5d07c57e09680f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.116959 24430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf8550c110746688d41e86305233931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.117067 24635 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.117578 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ce9769fcc4689babbf3d3f9de3d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:09.117831 24635 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.118171 24635 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.125504 24636 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.120594 24635 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.161790 24636 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.174407 24638 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.186672 24546 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.187053 24635 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.072s user 0.002s sys 0.012s
I20240426 13:18:09.188596 24635 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.195530 24636 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.196161 24636 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.009s
I20240426 13:18:09.199569 24635 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.207674 24638 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.200966 24636 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.208408 24636 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.208778 24636 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.209602 24635 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.209551 24636 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.210325 24635 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240426 13:18:09.215066 24638 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.215582 24638 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.002s
I20240426 13:18:09.217043 24635 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.218106 24635 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.218176 24638 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.219213 24638 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.219573 24638 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.220141 24620 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.218693 24635 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.220779 24636 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.024s user 0.015s sys 0.000s
I20240426 13:18:09.220896 24635 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.221792 24636 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.223476 24635 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240426 13:18:09.224314 24635 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.220314 24638 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.228085 24636 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.229943 24635 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.234757 24638 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.019s user 0.003s sys 0.009s
I20240426 13:18:09.235739 24471 heartbeater.cc:503] Master 127.23.61.62:42989 was elected leader, sending a full tablet report...
I20240426 13:18:09.241519 24635 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.242075 24635 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240426 13:18:09.242100 24638 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.244905 24635 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.245760 24635 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.246107 24635 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.246970 24635 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.250631 24638 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.250941 24635 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 13:18:09.251917 24635 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.257706 24635 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.266108 24636 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.266650 24636 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.020s
I20240426 13:18:09.266675 24638 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.267400 24638 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.018s
I20240426 13:18:09.268990 24635 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.269480 24635 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240426 13:18:09.269610 24636 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.270411 24636 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.270053 24638 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.270749 24636 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.270879 24638 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.271257 24638 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.271553 24636 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.272079 24638 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.272281 24635 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.273150 24635 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.273492 24635 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.273955 24636 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240426 13:18:09.274544 24638 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240426 13:18:09.274857 24636 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.274363 24635 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.275444 24638 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.278316 24635 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 13:18:09.279309 24635 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.280894 24636 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.281128 24638 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.285032 24635 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.291754 24638 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.292430 24638 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240426 13:18:09.295636 24638 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.301746 24638 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.301959 24636 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.302155 24638 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.302541 24636 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.008s
I20240426 13:18:09.302897 24638 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.305251 24638 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20240426 13:18:09.300983 24635 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.306006 24635 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.007s
I20240426 13:18:09.305613 24636 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.306201 24638 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.306486 24636 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.306912 24636 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.307737 24636 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.308701 24635 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.309562 24635 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.309895 24635 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.310331 24636 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240426 13:18:09.310716 24635 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.311314 24636 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.312804 24638 tablet_bootstrap.cc:654] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.315220 24635 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
I20240426 13:18:09.316170 24635 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.317414 24636 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.321439 24639 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.322039 24639 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.324429 24638 tablet_bootstrap.cc:492] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.324888 24638 ts_tablet_manager.cc:1367] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20240426 13:18:09.327409 24638 raft_consensus.cc:348] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.327881 24638 raft_consensus.cc:374] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.328250 24638 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.328359 24636 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.328822 24636 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240426 13:18:09.328837 24638 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.331459 24636 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.332386 24636 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.332789 24636 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.335631 24638 ts_tablet_manager.cc:1398] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240426 13:18:09.333693 24636 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.336740 24639 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:09.337234 24638 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.338625 24636 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240426 13:18:09.344879 24636 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.346769 24638 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.350418 24635 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.350546 24636 tablet_bootstrap.cc:654] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.352586 24639 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.356182 24639 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.358726 24639 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.361819 24639 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.367041 24638 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.367720 24638 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.008s
I20240426 13:18:09.364281 24639 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.370357 24639 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.372495 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.372627 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.374665 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.375552 24445 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.376683 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.377914 24594 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.380120 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
W20240426 13:18:09.380635 24486 leader_election.cc:341] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.381659 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.382267 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.382968 24593 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.382472 24484 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.384166 24636 tablet_bootstrap.cc:492] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.384579 24636 ts_tablet_manager.cc:1367] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.013s
I20240426 13:18:09.384596 24484 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.385551 24639 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.385898 24639 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.386345 24639 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.386374 24657 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.386848 24657 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.387056 24657 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.387087 24636 raft_consensus.cc:348] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.387847 24636 raft_consensus.cc:374] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.385937 24484 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.388234 24636 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
W20240426 13:18:09.389027 24486 leader_election.cc:341] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.389122 24636 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.391433 24636 ts_tablet_manager.cc:1398] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240426 13:18:09.393887 24644 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.394209 24638 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.394759 24636 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.394435 24644 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.394989 24638 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.395328 24638 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.397583 24638 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.400071 24644 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.412480 24657 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.413930 24643 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.414409 24643 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.414870 24635 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.415191 24657 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.415406 24635 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.009s
I20240426 13:18:09.416805 24643 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.417346 24644 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.417841 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.419368 24593 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.415926 24657 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.423754 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.424507 24444 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.427304 24636 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.417935 24644 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.430141 24644 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.430200 24593 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.431504 24484 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.422940 24657 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.435061 24657 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.435993 24444 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.436132 24660 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.436688 24635 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.444698 24639 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.456952 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.457813 24593 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.461592 24639 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:09.464784 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.465267 24638 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.097s user 0.011s sys 0.013s
I20240426 13:18:09.466306 24638 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
W20240426 13:18:09.467444 24486 leader_election.cc:341] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.469779 24662 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.470211 24662 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.460403 24657 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.473441 24662 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.474221 24657 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.475350 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.475937 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.478231 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.479305 24638 tablet_bootstrap.cc:654] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.446951 24635 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.485023 24521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.486410 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.487632 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
W20240426 13:18:09.482452 24486 leader_election.cc:341] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:09.489261 24411 leader_election.cc:341] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.471963 24593 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
W20240426 13:18:09.490135 24411 leader_election.cc:341] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:09.490892 24411 leader_election.cc:341] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.491014 24484 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.491744 24657 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.495961 24595 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.481495 24635 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.497347 24657 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.499261 24484 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.500365 24639 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.500061 24635 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.500775 24639 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.501502 24657 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.504207 24636 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.504689 24636 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.110s user 0.042s sys 0.016s
I20240426 13:18:09.501518 24639 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.507195 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.507369 24636 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.507819 24444 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.508129 24636 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.508466 24636 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.509351 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.509204 24636 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.510185 24643 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.510546 24643 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.510834 24643 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.517959 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.518994 24643 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.519449 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.498340 24660 raft_consensus.cc:686] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.520138 24594 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 0.
W20240426 13:18:09.526891 24409 leader_election.cc:341] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.527500 24409 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
W20240426 13:18:09.527760 24392 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:09.528728 24409 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:09.529446 24644 raft_consensus.cc:2726] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.529940 24662 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.530138 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.530303 24662 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.530668 24662 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.530966 24668 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.531430 24668 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.533748 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.534238 24643 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.536170 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.536509 24668 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.534330 24444 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:09.537547 24486 leader_election.cc:341] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.540733 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.541348 24594 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.542356 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.543427 24668 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.543816 24668 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.544224 24668 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.545779 24662 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.548413 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
W20240426 13:18:09.549640 24411 leader_election.cc:341] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.550559 24668 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.553112 24668 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.553318 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.553447 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.553864 24594 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:09.554862 24486 leader_election.cc:341] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.556535 24444 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
W20240426 13:18:09.557618 24409 leader_election.cc:341] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.557569 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.558125 24409 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.558697 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:09.528957 24660 consensus_queue.cc:237] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.559233 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.564224 24643 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.564718 24643 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:09.565521 24643 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.566696 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.567294 24595 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.567476 24636 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.062s user 0.015s sys 0.009s
I20240426 13:18:09.568166 24644 raft_consensus.cc:2726] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.568917 24636 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.569051 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.575052 24594 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.569386 24662 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.576339 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.578065 24657 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.578626 24657 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.579484 24657 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.580333 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.582103 24638 tablet_bootstrap.cc:492] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.582594 24638 ts_tablet_manager.cc:1367] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.116s user 0.019s sys 0.008s
I20240426 13:18:09.586370 24635 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.171s user 0.016s sys 0.013s
I20240426 13:18:09.586057 24342 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:09.587291 24635 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.575102 24595 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 1.
I20240426 13:18:09.588832 24409 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:09.591863 24638 raft_consensus.cc:348] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.592996 24638 raft_consensus.cc:374] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.593428 24638 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.594481 24638 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.595731 24662 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.596840 24636 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.600726 24635 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.601372 24662 raft_consensus.cc:686] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:09.604661 24662 consensus_queue.cc:237] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.603606 24638 ts_tablet_manager.cc:1398] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s
I20240426 13:18:09.608716 24638 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.609788 24635 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.610354 24635 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.002s
I20240426 13:18:09.620340 24635 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.621107 24635 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.621383 24635 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.622522 24638 tablet_bootstrap.cc:654] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.623396 24636 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.623773 24636 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.004s
I20240426 13:18:09.624275 24635 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.626438 24636 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.627380 24636 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.627689 24636 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.628549 24657 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.628988 24635 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.018s user 0.007s sys 0.003s
I20240426 13:18:09.628973 24657 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.630085 24635 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.630936 24657 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.631907 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.632299 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.633059 24636 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
W20240426 13:18:09.633695 24486 leader_election.cc:341] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.634002 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.636433 24635 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.637181 24484 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.638262 24657 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.638609 24657 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.638929 24657 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.640372 24344 catalog_manager.cc:5484] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.647361 24340 catalog_manager.cc:5484] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 1 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.650734 24636 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20240426 13:18:09.651964 24636 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.659325 24636 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.660248 24657 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.664126 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.664884 24657 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
W20240426 13:18:09.665987 24486 leader_election.cc:341] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:09.664211 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.666885 24595 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.675321 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.677276 24636 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.677822 24636 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.010s
I20240426 13:18:09.678397 24484 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters: 615118e943cc40268286df02ea3ee519
I20240426 13:18:09.679196 24657 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.679782 24657 raft_consensus.cc:686] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.680809 24657 consensus_queue.cc:237] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.682322 24638 tablet_bootstrap.cc:492] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.682739 24638 ts_tablet_manager.cc:1367] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.074s user 0.037s sys 0.024s
I20240426 13:18:09.685236 24638 raft_consensus.cc:348] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.685700 24638 raft_consensus.cc:374] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.685957 24638 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.686436 24635 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.686604 24638 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.686911 24636 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.688931 24638 ts_tablet_manager.cc:1398] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240426 13:18:09.691289 24635 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.038s
I20240426 13:18:09.691797 24638 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.693717 24635 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.694471 24635 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.695099 24635 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.698490 24638 tablet_bootstrap.cc:654] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.695756 24635 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.689527 24636 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.698900 24636 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.699540 24636 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.701515 24636 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240426 13:18:09.699987 24635 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 13:18:09.702462 24635 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.708557 24636 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.711007 24635 tablet_bootstrap.cc:654] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.711658 24638 tablet_bootstrap.cc:492] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.712155 24638 ts_tablet_manager.cc:1367] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20240426 13:18:09.714161 24636 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.715170 24638 raft_consensus.cc:348] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.715828 24638 raft_consensus.cc:374] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.716085 24638 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.716648 24638 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.718592 24638 ts_tablet_manager.cc:1398] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240426 13:18:09.720057 24638 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.719641 24344 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.721868 24344 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.725454 24638 tablet_bootstrap.cc:654] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.723587 24344 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.728080 24344 catalog_manager.cc:5484] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.731983 24636 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.732611 24636 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.008s
I20240426 13:18:09.734892 24635 tablet_bootstrap.cc:492] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.735280 24638 tablet_bootstrap.cc:492] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.735285 24635 ts_tablet_manager.cc:1367] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.013s
I20240426 13:18:09.735227 24636 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.735857 24638 ts_tablet_manager.cc:1367] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240426 13:18:09.736019 24636 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.736398 24636 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.737728 24635 raft_consensus.cc:348] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.738166 24635 raft_consensus.cc:374] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.738523 24635 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.738165 24638 raft_consensus.cc:348] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.737090 24636 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.738902 24638 raft_consensus.cc:374] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.739209 24638 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.739053 24635 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.739917 24638 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.740447 24636 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:18:09.741080 24635 ts_tablet_manager.cc:1398] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240426 13:18:09.741475 24636 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Bootstrap starting.
I20240426 13:18:09.741859 24635 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Bootstrap starting.
I20240426 13:18:09.745064 24638 ts_tablet_manager.cc:1398] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240426 13:18:09.745828 24638 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Bootstrap starting.
I20240426 13:18:09.747097 24635 tablet_bootstrap.cc:654] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.750900 24636 tablet_bootstrap.cc:654] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.756224 24638 tablet_bootstrap.cc:654] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:09.758471 24635 tablet_bootstrap.cc:492] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: No bootstrap required, opened a new log
I20240426 13:18:09.758881 24635 ts_tablet_manager.cc:1367] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240426 13:18:09.761305 24635 raft_consensus.cc:348] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.761966 24635 raft_consensus.cc:374] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.762213 24635 raft_consensus.cc:729] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.762859 24635 consensus_queue.cc:260] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.763448 24636 tablet_bootstrap.cc:492] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: No bootstrap required, opened a new log
I20240426 13:18:09.764169 24636 ts_tablet_manager.cc:1367] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20240426 13:18:09.765513 24635 ts_tablet_manager.cc:1398] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:09.767700 24636 raft_consensus.cc:348] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.768383 24636 raft_consensus.cc:374] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.768657 24636 raft_consensus.cc:729] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.769250 24636 consensus_queue.cc:260] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.771152 24636 ts_tablet_manager.cc:1398] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240426 13:18:09.771831 24638 tablet_bootstrap.cc:492] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: No bootstrap required, opened a new log
I20240426 13:18:09.772276 24638 ts_tablet_manager.cc:1367] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.003s
I20240426 13:18:09.774896 24638 raft_consensus.cc:348] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.775534 24638 raft_consensus.cc:374] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:09.775797 24638 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Initialized, Role: FOLLOWER
I20240426 13:18:09.776346 24638 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.777889 24638 ts_tablet_manager.cc:1398] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240426 13:18:09.785792 24657 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.786157 24657 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.787734 24657 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.788341 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.788650 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.788952 24444 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.789222 24595 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.789901 24486 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:09.790591 24657 raft_consensus.cc:2781] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.790895 24657 raft_consensus.cc:482] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.791152 24657 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.795398 24657 raft_consensus.cc:504] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.797232 24657 leader_election.cc:288] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.797657 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.797773 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ce9769fcc4689babbf3d3f9de3d57" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.798100 24444 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.798226 24595 raft_consensus.cc:3037] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.802213 24444 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.802593 24595 raft_consensus.cc:2454] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.803050 24486 leader_election.cc:302] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:09.803748 24657 raft_consensus.cc:2781] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.804137 24657 raft_consensus.cc:686] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.804713 24657 consensus_queue.cc:237] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.811292 24343 catalog_manager.cc:5484] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.842130 24643 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.842592 24643 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.844118 24643 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.844971 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.845068 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.845464 24444 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.845587 24520 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 0.
I20240426 13:18:09.846278 24560 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.846877 24643 raft_consensus.cc:2781] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.847141 24643 raft_consensus.cc:482] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.847384 24643 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.851436 24643 raft_consensus.cc:504] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.852758 24643 leader_election.cc:288] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.853446 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.853628 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdcbd8d33f24437ba5d07c57e09680f" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:09.853878 24444 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.854102 24520 raft_consensus.cc:3037] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.857920 24444 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.857961 24520 raft_consensus.cc:2454] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 1.
I20240426 13:18:09.858786 24560 leader_election.cc:302] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:09.859344 24643 raft_consensus.cc:2781] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.859699 24643 raft_consensus.cc:686] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:09.860435 24643 consensus_queue.cc:237] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.866593 24343 catalog_manager.cc:5484] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:09.880455 24662 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.880861 24662 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.882242 24662 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.883097 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.883008 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.883901 24595 raft_consensus.cc:2379] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 615118e943cc40268286df02ea3ee519 in current term 1: Already voted for candidate 9d0a6220c3eb4062b0365da45223700d in this term.
I20240426 13:18:09.884760 24409 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.885308 24662 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.889380 24662 raft_consensus.cc:2726] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.924078 24657 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.924553 24657 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.926090 24657 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.926822 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.927055 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.927474 24444 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.927588 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 0.
I20240426 13:18:09.928504 24486 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:09.929098 24657 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:09.929351 24657 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:09.929591 24657 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.933810 24657 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.935252 24657 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.935971 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:09.936146 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:09.936477 24444 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.936616 24595 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.940594 24444 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.941002 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 1.
I20240426 13:18:09.941525 24486 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:09.942277 24657 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:09.942631 24657 raft_consensus.cc:686] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:09.943362 24657 consensus_queue.cc:237] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.949604 24343 catalog_manager.cc:5484] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 1 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:09.959749 24662 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.960172 24662 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:09.961649 24662 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:09.962337 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.962605 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0a6220c3eb4062b0365da45223700d" is_pre_election: true
I20240426 13:18:09.962944 24595 raft_consensus.cc:2379] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 615118e943cc40268286df02ea3ee519 in current term 1: Already voted for candidate 9d0a6220c3eb4062b0365da45223700d in this term.
I20240426 13:18:09.963798 24409 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 615118e943cc40268286df02ea3ee519; no voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.964414 24662 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:09.968488 24662 raft_consensus.cc:2726] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:09.994741 24657 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:09.995182 24657 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:09.996703 24657 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:09.997684 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" is_pre_election: true
I20240426 13:18:09.997897 24444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615118e943cc40268286df02ea3ee519" is_pre_election: true
I20240426 13:18:09.998284 24595 raft_consensus.cc:2379] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d0a6220c3eb4062b0365da45223700d in current term 1: Already voted for candidate 615118e943cc40268286df02ea3ee519 in this term.
I20240426 13:18:09.999174 24484 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d; no voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:09.999892 24657 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.001950 24660 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.005141 24657 raft_consensus.cc:2726] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:10.013494 24657 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.031818 24657 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.040792 24670 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.062981 24444 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.064364 24660 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.070236 24660 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.074573 24670 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.082999 24446 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:10.088748 24668 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.098644 24668 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.109489 24660 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.115645 24643 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.140651 24643 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:10.145081 24519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.146026 24519 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.165841 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:10.166289 24595 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.170367 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.170780 24520 raft_consensus.cc:595] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.174588 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:10.175026 24520 raft_consensus.cc:595] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:10.188056 24662 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.198786 24718 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.320930 24714 consensus_queue.cc:1035] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.330791 24714 consensus_queue.cc:1035] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.331158 24660 consensus_queue.cc:1035] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.343453 24668 consensus_queue.cc:1035] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.362241 24668 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.371907 24657 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:10.379263 24657 raft_consensus.cc:979] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.379751 24657 raft_consensus.cc:1067] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.381240 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.381721 24445 raft_consensus.cc:482] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.381994 24445 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.386639 24445 raft_consensus.cc:504] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.388231 24445 leader_election.cc:288] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.388914 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.389230 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc669f73c8d44f49ddd7df04d27bce7" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.389523 24520 raft_consensus.cc:3032] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.389717 24595 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.389906 24520 raft_consensus.cc:729] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.390697 24520 consensus_queue.cc:260] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.391970 24520 raft_consensus.cc:3037] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.394752 24595 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.395709 24409 leader_election.cc:302] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:10.396462 24718 raft_consensus.cc:2781] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.396890 24718 raft_consensus.cc:686] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.397209 24520 raft_consensus.cc:2454] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.397612 24718 consensus_queue.cc:237] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.403208 24342 catalog_manager.cc:5484] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.559386 24668 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.559813 24657 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.561215 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.561741 24445 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.562111 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.566792 24445 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.568365 24445 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:10.568876 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.569278 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.569644 24595 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.569753 24520 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.570057 24520 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.570634 24520 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.571516 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.575022 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.575611 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.575986 24409 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:10.576733 24718 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.577109 24718 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.577716 24718 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:10.584347 24342 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:10.629823 24657 raft_consensus.cc:979] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.630251 24657 raft_consensus.cc:1067] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.631603 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:10.632161 24445 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.632544 24445 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.636821 24445 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.638070 24445 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.638852 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.638979 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.639343 24520 raft_consensus.cc:3032] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.639461 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.639640 24520 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:10.640322 24520 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.641438 24520 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.645851 24643 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.646262 24714 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:10.646535 24520 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.646960 24595 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.647437 24411 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:10.647687 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:10.648239 24718 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.648365 24445 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:10.648648 24718 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.648770 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.649426 24718 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.653869 24445 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.655990 24445 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:10.655438 24342 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.656229 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:10.656934 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.656833 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:10.657418 24595 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:10.657718 24595 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:10.658403 24595 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.659564 24595 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:10.663019 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.663971 24411 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:10.664644 24718 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:10.665110 24718 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:10.666043 24595 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:10.665992 24718 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:10.671600 24342 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:10.927685 24520 raft_consensus.cc:1261] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:10.928807 24718 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:10.937393 24595 raft_consensus.cc:1261] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:10.938670 24718 consensus_queue.cc:1035] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.071800 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.073516 24719 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.074016 24520 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.073958 24595 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.075450 24719 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.075865 24718 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:11.087199 24519 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.095615 24595 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.099464 24595 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:11.100730 24719 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:18:11.104676 24732 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:11.105891 24718 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:13.208653 23796 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240426 13:18:13.209298 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:13.224800 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:13.225255 24445 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:13.227192 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:13.227594 24445 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:13.228441 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240426 13:18:13.228821 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:13.582510 24719 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.582907 24719 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.584195 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:13.584620 24594 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:13.584878 24594 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.589005 24594 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.590376 24594 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:13.591198 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:13.591296 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:13.591856 24519 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.591885 24445 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:13.592396 24445 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:13.593106 24445 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.594417 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.598174 24519 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.599573 24560 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:13.600337 24672 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:13.600582 24445 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.601294 24672 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:13.602145 24672 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:13.609355 24343 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:13.635968 24739 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.636442 24719 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:13.637728 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:13.638198 24594 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:13.638473 24594 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.642700 24594 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.644145 24594 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:13.645009 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:13.645126 24519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:13.645649 24445 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:13.645704 24519 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.646097 24445 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:13.646862 24445 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.648146 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:13.652216 24519 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.653234 24560 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:13.653889 24672 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:13.654012 24445 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:13.654302 24672 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:13.655169 24672 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:13.661741 24343 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:13.997252 24445 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:13.998425 24672 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.006915 24519 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.008442 24672 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.153931 24519 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.154987 24672 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.162557 24445 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:14.164521 24740 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:14.229691 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:14.234771 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:14.235152 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:15.235841 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:15.241791 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:15.242154 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:16.242888 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:16.247956 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:16.248363 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:17.249097 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:17.255771 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:17.256155 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:18.257117 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:18.262763 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:18.263126 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:19.263839 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:19.269687 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:19.270059 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:20.270879 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:20.275799 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:20.276223 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:21.277122 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:21.282230 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:21.282624 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:22.283318 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:22.289212 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:22.289563 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:23.311794 23796 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240426 13:18:23.316251 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:23.316884 24595 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.322889 24595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:53306
I20240426 13:18:23.323289 24595 raft_consensus.cc:595] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.331246 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.331801 24520 raft_consensus.cc:595] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.335915 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.336381 24520 raft_consensus.cc:595] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.340257 24520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
mode: GRACEFUL
new_leader_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:41906
I20240426 13:18:23.340785 24520 raft_consensus.cc:595] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Received request to transfer leadership to TS 615118e943cc40268286df02ea3ee519
I20240426 13:18:23.343062 24761 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.343607 24764 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.345183 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:23.345618 24445 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.345944 24445 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.351711 24445 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.353560 24445 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Requested vote from peers f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:23.354269 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.354519 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.355145 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.355159 24595 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:18:23.355583 24595 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:23.356215 24595 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.357115 24595 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.359797 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.360937 24411 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:23.361765 24762 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:18:23.362149 24595 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.362782 24762 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.363649 24762 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:23.369774 24343 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 3 to 4, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 4 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:23.386794 24757 raft_consensus.cc:979] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.387153 24757 raft_consensus.cc:1067] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.388252 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.388655 24445 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.388911 24445 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.393175 24445 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.394405 24445 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.395146 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.395236 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.395684 24520 raft_consensus.cc:3032] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.395721 24595 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.396095 24520 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.396718 24520 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.397747 24520 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.399739 24595 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.400589 24409 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:23.401198 24762 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.401516 24762 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.402290 24762 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.402563 24520 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.408191 24343 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.524778 24771 raft_consensus.cc:979] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.525189 24757 raft_consensus.cc:1067] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.526356 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.526706 24445 raft_consensus.cc:482] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.526902 24445 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.531055 24445 raft_consensus.cc:504] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.532727 24445 leader_election.cc:288] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.532756 24761 raft_consensus.cc:979] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.533084 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.533383 24761 raft_consensus.cc:1067] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.533653 24520 raft_consensus.cc:3032] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.533522 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f0e49eace343cfaef009c9f693fa3b" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.534044 24520 raft_consensus.cc:729] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.534237 24595 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.534785 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48504
I20240426 13:18:23.534790 24520 consensus_queue.cc:260] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.535408 24445 raft_consensus.cc:482] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.535760 24445 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.535916 24520 raft_consensus.cc:3037] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.539994 24595 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.540161 24520 raft_consensus.cc:2454] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.540997 24409 leader_election.cc:302] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:23.541409 24445 raft_consensus.cc:504] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.541615 24762 raft_consensus.cc:2781] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.542105 24762 raft_consensus.cc:686] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.542909 24445 leader_election.cc:288] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.542865 24762 consensus_queue.cc:237] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.543870 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.544075 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350efe2144740e7bb3f2380a2a3ed85" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.544415 24520 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.544525 24595 raft_consensus.cc:3032] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:18:23.548221 24595 raft_consensus.cc:729] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:23.548969 24595 consensus_queue.cc:260] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.550002 24520 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.550155 24595 raft_consensus.cc:3037] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:18:23.550536 24343 catalog_manager.cc:5484] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.552769 24411 leader_election.cc:302] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, 9d0a6220c3eb4062b0365da45223700d; no voters:
I20240426 13:18:23.553547 24762 raft_consensus.cc:2781] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:18:23.554190 24762 raft_consensus.cc:686] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.555305 24762 consensus_queue.cc:237] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.555636 24595 raft_consensus.cc:2454] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 4.
I20240426 13:18:23.561604 24343 catalog_manager.cc:5484] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 3 to 4, leader changed from f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 4 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.583031 24771 raft_consensus.cc:979] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: : Instructing follower 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.583482 24757 raft_consensus.cc:1067] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Signalling peer 615118e943cc40268286df02ea3ee519 to start an election
I20240426 13:18:23.584676 24445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6"
dest_uuid: "615118e943cc40268286df02ea3ee519"
from {username='slave'} at 127.0.0.1:48496
I20240426 13:18:23.585129 24445 raft_consensus.cc:482] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:23.585480 24445 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.589217 24445 raft_consensus.cc:504] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.590251 24445 leader_election.cc:288] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:23.590955 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:23.591090 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ef08e079494079bb32b9555a38e8a6" candidate_uuid: "615118e943cc40268286df02ea3ee519" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:23.591383 24520 raft_consensus.cc:3032] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:23.591523 24595 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.591645 24520 raft_consensus.cc:729] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:23.592348 24520 consensus_queue.cc:260] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.593183 24520 raft_consensus.cc:3037] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:23.595335 24595 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.596082 24409 leader_election.cc:302] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615118e943cc40268286df02ea3ee519, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:23.596669 24762 raft_consensus.cc:2781] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:23.596738 24520 raft_consensus.cc:2454] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615118e943cc40268286df02ea3ee519 in term 2.
I20240426 13:18:23.597034 24762 raft_consensus.cc:686] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Leader. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:23.597955 24762 consensus_queue.cc:237] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:23.604017 24343 catalog_manager.cc:5484] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 reported cstate change: term changed from 1 to 2, leader changed from 9d0a6220c3eb4062b0365da45223700d (127.23.61.2) to 615118e943cc40268286df02ea3ee519 (127.23.61.1). New cstate: current_term: 2 leader_uuid: "615118e943cc40268286df02ea3ee519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:23.791553 24595 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.792573 24762 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.802284 24520 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.804806 24779 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.815157 24595 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.816138 24779 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:23.823470 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.825613 24762 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:23.961488 24595 raft_consensus.cc:1261] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.962568 24762 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.970515 24520 raft_consensus.cc:1261] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:23.971843 24762 consensus_queue.cc:1035] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:23.994836 24520 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:23.995630 24762 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:24.005477 24595 raft_consensus.cc:1261] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:18:24.006863 24787 consensus_queue.cc:1035] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:24.009928 24520 raft_consensus.cc:1261] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:24.011371 24787 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:24.022145 24594 raft_consensus.cc:1261] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 615118e943cc40268286df02ea3ee519: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:24.024140 24762 consensus_queue.cc:1035] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:26.368892 23796 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240426 13:18:26.369436 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:26.376086 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "9d0a6220c3eb4062b0365da45223700d"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.376513 24445 raft_consensus.cc:595] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Received request to transfer leadership to TS 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:26.378425 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.378773 24445 raft_consensus.cc:595] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:26.380465 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.380836 24445 raft_consensus.cc:595] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS f6a9d1d3ef9d41489d7ab9a93c740798
I20240426 13:18:26.382557 24445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "615118e943cc40268286df02ea3ee519"
mode: GRACEFUL
new_leader_uuid: "9d0a6220c3eb4062b0365da45223700d"
from {username='slave'} at 127.0.0.1:53944
I20240426 13:18:26.382889 24445 raft_consensus.cc:595] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Received request to transfer leadership to TS 9d0a6220c3eb4062b0365da45223700d
I20240426 13:18:26.383544 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:18:26.383893 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:26.560498 24793 raft_consensus.cc:979] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: : Instructing follower 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.560904 24787 raft_consensus.cc:1067] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.562018 24520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cbf8550c110746688d41e86305233931"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
from {username='slave'} at 127.0.0.1:41888
I20240426 13:18:26.562418 24520 raft_consensus.cc:482] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.562741 24520 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.565198 24793 raft_consensus.cc:979] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.565570 24787 raft_consensus.cc:1067] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.566702 24594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:26.567184 24594 raft_consensus.cc:482] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.567495 24594 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.567891 24520 raft_consensus.cc:504] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.570231 24520 leader_election.cc:288] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:26.570966 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:26.570966 24445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf8550c110746688d41e86305233931" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.571664 24595 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.571673 24445 raft_consensus.cc:3032] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:26.572223 24445 raft_consensus.cc:729] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.572909 24445 consensus_queue.cc:260] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.573076 24594 raft_consensus.cc:504] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.574041 24445 raft_consensus.cc:3037] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.575239 24594 leader_election.cc:288] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 5 election: Requested vote from peers 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815), 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527)
I20240426 13:18:26.576020 24595 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 3.
I20240426 13:18:26.576788 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.577287 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445fcfc9676148b99b04c11e1724fe62" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:26.577323 24446 raft_consensus.cc:3032] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:18:26.577245 24484 leader_election.cc:302] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:26.577937 24446 raft_consensus.cc:729] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.578397 24757 raft_consensus.cc:2781] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:26.578686 24446 consensus_queue.cc:260] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.579495 24445 raft_consensus.cc:2454] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 3.
I20240426 13:18:26.580284 24446 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.580570 24757 raft_consensus.cc:686] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:26.581323 24757 consensus_queue.cc:237] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.577934 24520 raft_consensus.cc:3037] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:18:26.586889 24520 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 5.
I20240426 13:18:26.587241 24446 raft_consensus.cc:2454] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 5.
I20240426 13:18:26.587873 24560 leader_election.cc:302] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:26.588810 24792 raft_consensus.cc:2781] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:18:26.590252 24792 raft_consensus.cc:686] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:26.590982 24792 consensus_queue.cc:237] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } }
I20240426 13:18:26.591233 24343 catalog_manager.cc:5484] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 3 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.598729 24342 catalog_manager.cc:5484] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 4 to 5, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.703052 24793 raft_consensus.cc:979] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: : Instructing follower 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.703416 24787 raft_consensus.cc:1067] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Signalling peer 9d0a6220c3eb4062b0365da45223700d to start an election
I20240426 13:18:26.704604 24520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398"
dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
from {username='slave'} at 127.0.0.1:41888
I20240426 13:18:26.705041 24520 raft_consensus.cc:482] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.705307 24520 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.709611 24520 raft_consensus.cc:504] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.710845 24520 leader_election.cc:288] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 2 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3:45341)
I20240426 13:18:26.711594 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.711720 24595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27eaa94161a44b1ab098df62ef5f5398" candidate_uuid: "9d0a6220c3eb4062b0365da45223700d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
I20240426 13:18:26.712291 24446 raft_consensus.cc:3032] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:18:26.712342 24595 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.712574 24446 raft_consensus.cc:729] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.713322 24446 consensus_queue.cc:260] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.714378 24446 raft_consensus.cc:3037] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:18:26.716688 24595 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 2.
I20240426 13:18:26.717475 24484 leader_election.cc:302] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:26.718081 24757 raft_consensus.cc:2781] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:18:26.718400 24757 raft_consensus.cc:686] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 LEADER]: Becoming Leader. State: Replica: 9d0a6220c3eb4062b0365da45223700d, State: Running, Role: LEADER
I20240426 13:18:26.718822 24446 raft_consensus.cc:2454] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a6220c3eb4062b0365da45223700d in term 2.
I20240426 13:18:26.718963 24757 consensus_queue.cc:237] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.725360 24342 catalog_manager.cc:5484] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d reported cstate change: term changed from 1 to 2, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to 9d0a6220c3eb4062b0365da45223700d (127.23.61.2). New cstate: current_term: 2 leader_uuid: "9d0a6220c3eb4062b0365da45223700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:26.771771 24793 raft_consensus.cc:979] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: : Instructing follower f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.772209 24787 raft_consensus.cc:1067] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Signalling peer f6a9d1d3ef9d41489d7ab9a93c740798 to start an election
I20240426 13:18:26.773352 24595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "962e435412bd4d65b99b276341083d43"
dest_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798"
from {username='slave'} at 127.0.0.1:53294
I20240426 13:18:26.773756 24595 raft_consensus.cc:482] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:18:26.774019 24595 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.777953 24595 raft_consensus.cc:504] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.779239 24595 leader_election.cc:288] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Requested vote from peers 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527), 9d0a6220c3eb4062b0365da45223700d (127.23.61.2:33815)
I20240426 13:18:26.779961 24446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "615118e943cc40268286df02ea3ee519"
I20240426 13:18:26.780230 24520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962e435412bd4d65b99b276341083d43" candidate_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9d0a6220c3eb4062b0365da45223700d"
I20240426 13:18:26.780460 24446 raft_consensus.cc:3032] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:18:26.780681 24520 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.780776 24446 raft_consensus.cc:729] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 615118e943cc40268286df02ea3ee519, State: Running, Role: LEADER
I20240426 13:18:26.781524 24446 consensus_queue.cc:260] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.782426 24446 raft_consensus.cc:3037] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:18:26.784751 24520 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:26.785586 24560 leader_election.cc:302] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a6220c3eb4062b0365da45223700d, f6a9d1d3ef9d41489d7ab9a93c740798; no voters:
I20240426 13:18:26.786203 24792 raft_consensus.cc:2781] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:18:26.786659 24792 raft_consensus.cc:686] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Becoming Leader. State: Replica: f6a9d1d3ef9d41489d7ab9a93c740798, State: Running, Role: LEADER
I20240426 13:18:26.786937 24446 raft_consensus.cc:2454] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a9d1d3ef9d41489d7ab9a93c740798 in term 3.
I20240426 13:18:26.787550 24792 consensus_queue.cc:237] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } }
I20240426 13:18:26.793169 24342 catalog_manager.cc:5484] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 reported cstate change: term changed from 2 to 3, leader changed from 615118e943cc40268286df02ea3ee519 (127.23.61.1) to f6a9d1d3ef9d41489d7ab9a93c740798 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:26.984022 24595 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:26.985570 24757 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:26.994307 24446 raft_consensus.cc:1261] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:26.996204 24757 consensus_queue.cc:1035] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.046972 24520 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:18:27.048007 24792 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:18:27.055377 24446 raft_consensus.cc:1261] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:18:27.056818 24810 consensus_queue.cc:1035] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:18:27.148497 24446 raft_consensus.cc:1261] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:27.149484 24757 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:27.156728 24595 raft_consensus.cc:1261] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Refusing update from remote peer 9d0a6220c3eb4062b0365da45223700d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:18:27.158149 24813 consensus_queue.cc:1035] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a9d1d3ef9d41489d7ab9a93c740798" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:27.292424 24520 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:27.293480 24810 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0a6220c3eb4062b0365da45223700d" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.301693 24446 raft_consensus.cc:1261] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Refusing update from remote peer f6a9d1d3ef9d41489d7ab9a93c740798: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:18:27.303467 24810 consensus_queue.cc:1035] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615118e943cc40268286df02ea3ee519" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:27.384740 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:27.390419 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:27.390812 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:28.391917 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:28.399250 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:28.399643 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:29.400456 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:29.407269 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:29.407652 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:30.408588 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:30.414304 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:30.414804 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:31.415517 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:31.420588 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:31.421066 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:32.421937 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:32.427043 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:32.427412 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:33.428278 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:33.433764 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:33.434232 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:34.435015 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:34.440019 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:34.440415 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:35.441082 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:18:35.445895 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240426 13:18:35.446224 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:18:36.467764 23796 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240426 13:18:36.468273 23796 tablet_server.cc:178] TabletServer@127.23.61.1:0 shutting down...
W20240426 13:18:36.480716 24560 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20240426 13:18:36.484844 24560 consensus_peers.cc:485] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:36.495002 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.495594 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.496347 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.496842 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.498685 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.499121 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.499514 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P 615118e943cc40268286df02ea3ee519 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.501117 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.501559 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.502264 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.503873 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.504366 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.504737 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P 615118e943cc40268286df02ea3ee519 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.506340 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.506794 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.507529 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.509121 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.509547 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.509894 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.511482 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.511910 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.512351 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.514443 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.514966 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.515626 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P 615118e943cc40268286df02ea3ee519 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.517555 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.518245 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.519232 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P 615118e943cc40268286df02ea3ee519 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.521179 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519: stopping tablet replica
I20240426 13:18:36.521652 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.522013 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P 615118e943cc40268286df02ea3ee519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.550900 23796 tablet_server.cc:194] TabletServer@127.23.61.1:0 shutdown complete.
I20240426 13:18:36.568535 23796 tablet_server.cc:178] TabletServer@127.23.61.2:0 shutting down...
W20240426 13:18:36.583885 24486 consensus_peers.cc:485] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:36.586889 24560 consensus_peers.cc:485] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 -> Peer 615118e943cc40268286df02ea3ee519 (127.23.61.1:44527): Couldn't send request to peer 615118e943cc40268286df02ea3ee519. Status: Network error: Client connection negotiation failed: client connection to 127.23.61.1:44527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:36.601452 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.602129 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.603081 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.604069 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.605715 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.606124 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.606454 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.607894 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.608330 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.608952 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.610339 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.610819 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.611155 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.612932 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.613380 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.613842 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P 9d0a6220c3eb4062b0365da45223700d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.615521 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.615931 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.616333 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.617798 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.618211 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.618566 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P 9d0a6220c3eb4062b0365da45223700d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.620173 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.620620 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.621031 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P 9d0a6220c3eb4062b0365da45223700d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.622705 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.623172 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.623528 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P 9d0a6220c3eb4062b0365da45223700d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.624998 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d: stopping tablet replica
I20240426 13:18:36.625375 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.626062 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P 9d0a6220c3eb4062b0365da45223700d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.652815 23796 tablet_server.cc:194] TabletServer@127.23.61.2:0 shutdown complete.
I20240426 13:18:36.669242 23796 tablet_server.cc:178] TabletServer@127.23.61.3:0 shutting down...
I20240426 13:18:36.691056 23796 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240426 13:18:36.691653 23796 tablet_replica.cc:330] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.692263 23796 raft_consensus.cc:2229] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.692701 23796 raft_consensus.cc:2258] T e63ce9769fcc4689babbf3d3f9de3d57 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.694380 23796 tablet_replica.cc:330] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.694859 23796 raft_consensus.cc:2229] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.695268 23796 raft_consensus.cc:2258] T 6dc669f73c8d44f49ddd7df04d27bce7 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.696763 23796 tablet_replica.cc:330] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.697222 23796 raft_consensus.cc:2229] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.697584 23796 raft_consensus.cc:2258] T e4ef08e079494079bb32b9555a38e8a6 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.698976 23796 tablet_replica.cc:330] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.699406 23796 raft_consensus.cc:2229] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.699774 23796 raft_consensus.cc:2258] T cbf8550c110746688d41e86305233931 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.701236 23796 tablet_replica.cc:330] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.701679 23796 raft_consensus.cc:2229] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.702355 23796 raft_consensus.cc:2258] T 445fcfc9676148b99b04c11e1724fe62 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.703748 23796 tablet_replica.cc:330] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.704244 23796 raft_consensus.cc:2229] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.704933 23796 raft_consensus.cc:2258] T 9fdcbd8d33f24437ba5d07c57e09680f P f6a9d1d3ef9d41489d7ab9a93c740798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.706432 23796 tablet_replica.cc:330] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.706889 23796 raft_consensus.cc:2229] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.707235 23796 raft_consensus.cc:2258] T 9350efe2144740e7bb3f2380a2a3ed85 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.708731 23796 tablet_replica.cc:330] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.709126 23796 raft_consensus.cc:2229] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.709465 23796 raft_consensus.cc:2258] T 14f0e49eace343cfaef009c9f693fa3b P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.710916 23796 tablet_replica.cc:330] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.711352 23796 raft_consensus.cc:2229] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.712158 23796 raft_consensus.cc:2258] T 962e435412bd4d65b99b276341083d43 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.713629 23796 tablet_replica.cc:330] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798: stopping tablet replica
I20240426 13:18:36.714089 23796 raft_consensus.cc:2229] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:36.714416 23796 raft_consensus.cc:2258] T 27eaa94161a44b1ab098df62ef5f5398 P f6a9d1d3ef9d41489d7ab9a93c740798 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.741904 23796 tablet_server.cc:194] TabletServer@127.23.61.3:0 shutdown complete.
I20240426 13:18:36.758637 23796 master.cc:537] Master@127.23.61.62:42989 shutting down...
I20240426 13:18:36.783674 23796 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 LEADER]: Raft consensus shutting down.
I20240426 13:18:36.784241 23796 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:36.784546 23796 tablet_replica.cc:330] T 00000000000000000000000000000000 P 947ecbe50c614eaebc7282c4f49c3f4d: stopping tablet replica
I20240426 13:18:36.802116 23796 master.cc:552] Master@127.23.61.62:42989 shutdown complete.
615118e943cc40268286df02ea3ee519 5
9d0a6220c3eb4062b0365da45223700d 4
f6a9d1d3ef9d41489d7ab9a93c740798 1
615118e943cc40268286df02ea3ee519 3
9d0a6220c3eb4062b0365da45223700d 4
f6a9d1d3ef9d41489d7ab9a93c740798 3
615118e943cc40268286df02ea3ee519 8
9d0a6220c3eb4062b0365da45223700d 1
f6a9d1d3ef9d41489d7ab9a93c740798 1
615118e943cc40268286df02ea3ee519 4
9d0a6220c3eb4062b0365da45223700d 3
f6a9d1d3ef9d41489d7ab9a93c740798 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28452 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240426 13:18:36.842198 23796 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.61.62:34153
I20240426 13:18:36.843310 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:36.848953 24830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:36.849054 24829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:36.849535 24832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:36.851233 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:36.852423 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:36.852622 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:36.852799 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137516852781 us; error 0 us; skew 500 ppm
I20240426 13:18:36.853384 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:36.860855 23796 webserver.cc:460] Webserver started at http://127.23.61.62:34839/ using document root <none> and password file <none>
I20240426 13:18:36.861356 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:36.861528 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:36.861795 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:36.862982 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root/instance:
uuid: "e2ab89ff7e18462eb10e19320ff8beaa"
format_stamp: "Formatted at 2024-04-26 13:18:36 on dist-test-slave-jx77"
I20240426 13:18:36.867147 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240426 13:18:36.870378 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:36.871083 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 13:18:36.871344 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
uuid: "e2ab89ff7e18462eb10e19320ff8beaa"
format_stamp: "Formatted at 2024-04-26 13:18:36 on dist-test-slave-jx77"
I20240426 13:18:36.871628 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:36.896713 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:36.897953 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:36.933480 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.62:34153
I20240426 13:18:36.933584 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.62:34153 every 8 connection(s)
I20240426 13:18:36.937157 24889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:36.946286 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Bootstrap starting.
I20240426 13:18:36.950181 24889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:36.953986 24889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: No bootstrap required, opened a new log
I20240426 13:18:36.955850 24889 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.956326 24889 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:36.956614 24889 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ab89ff7e18462eb10e19320ff8beaa, State: Initialized, Role: FOLLOWER
I20240426 13:18:36.957108 24889 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.957564 24889 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 13:18:36.957779 24889 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 13:18:36.958047 24889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:36.961709 24889 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.962178 24889 leader_election.cc:302] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ab89ff7e18462eb10e19320ff8beaa; no voters:
I20240426 13:18:36.963203 24889 leader_election.cc:288] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240426 13:18:36.963474 24892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:36.964784 24892 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [term 1 LEADER]: Becoming Leader. State: Replica: e2ab89ff7e18462eb10e19320ff8beaa, State: Running, Role: LEADER
I20240426 13:18:36.965428 24892 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER }
I20240426 13:18:36.965955 24889 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: configured and running, proceeding with master startup.
I20240426 13:18:36.970970 24894 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ab89ff7e18462eb10e19320ff8beaa. Latest consensus state: current_term: 1 leader_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER } }
I20240426 13:18:36.971554 24894 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:36.972329 24893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab89ff7e18462eb10e19320ff8beaa" member_type: VOTER } }
I20240426 13:18:36.972930 24893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa [sys.catalog]: This master's current role is: LEADER
I20240426 13:18:36.974047 24902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 13:18:36.981303 24902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 13:18:36.982208 23796 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 13:18:36.989519 24902 catalog_manager.cc:1347] Generated new cluster ID: 7722385be3804d228a41d0506276be2b
I20240426 13:18:36.989749 24902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 13:18:37.010066 24902 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 13:18:37.011257 24902 catalog_manager.cc:1504] Loading token signing keys...
I20240426 13:18:37.022332 24902 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Generated new TSK 0
I20240426 13:18:37.022873 24902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 13:18:37.048333 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.053784 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.054636 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.055977 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.056763 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:37.057502 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.057677 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.057793 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517057783 us; error 0 us; skew 500 ppm
I20240426 13:18:37.058274 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.060441 23796 webserver.cc:460] Webserver started at http://127.23.61.1:35125/ using document root <none> and password file <none>
I20240426 13:18:37.060858 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.061013 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.061266 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.062302 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/instance:
uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.066541 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240426 13:18:37.069633 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.070298 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240426 13:18:37.070556 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.070799 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.094377 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:37.095510 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.096913 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.099117 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.099287 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.099519 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.099678 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.140969 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.1:41817
I20240426 13:18:37.141070 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.1:41817 every 8 connection(s)
I20240426 13:18:37.145283 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.152326 24985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.153764 24986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.156459 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:37.157829 24988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.158686 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.158910 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.159071 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517159056 us; error 0 us; skew 500 ppm
I20240426 13:18:37.159791 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.159787 24981 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.160257 24981 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.160998 24981 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.162374 23796 webserver.cc:460] Webserver started at http://127.23.61.2:39467/ using document root <none> and password file <none>
I20240426 13:18:37.163033 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.163290 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.163198 24854 ts_manager.cc:194] Registered new tserver with Master: 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:37.163679 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.165220 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/instance:
uuid: "46106ff992984d33bb4a4d8f30162639"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.165660 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39836
I20240426 13:18:37.169842 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240426 13:18:37.173144 24993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.173852 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240426 13:18:37.174106 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
uuid: "46106ff992984d33bb4a4d8f30162639"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.174358 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.195663 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:37.196897 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.198227 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.200433 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.200621 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.200835 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.201010 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.237288 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.2:43671
I20240426 13:18:37.237373 25055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.2:43671 every 8 connection(s)
I20240426 13:18:37.241653 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:37.247750 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.249310 25060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 13:18:37.250761 25062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:37.251791 23796 server_base.cc:1034] running on GCE node
I20240426 13:18:37.252781 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:37.253001 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:37.253172 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137517253146 us; error 0 us; skew 500 ppm
I20240426 13:18:37.253758 25056 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.253752 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:37.254179 25056 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.254832 25056 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.256438 23796 webserver.cc:460] Webserver started at http://127.23.61.3:45639/ using document root <none> and password file <none>
I20240426 13:18:37.256829 24854 ts_manager.cc:194] Registered new tserver with Master: 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:37.256907 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:37.257206 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:37.257457 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:37.258100 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39848
I20240426 13:18:37.258579 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/instance:
uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.262818 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240426 13:18:37.265904 25067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.266606 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240426 13:18:37.266903 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
format_stamp: "Formatted at 2024-04-26 13:18:37 on dist-test-slave-jx77"
I20240426 13:18:37.267154 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:37.288144 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:37.289194 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:37.290521 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:37.292704 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:37.292876 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.293118 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:37.293255 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:37.329595 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.3:38089
I20240426 13:18:37.329687 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.3:38089 every 8 connection(s)
I20240426 13:18:37.341786 25130 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:37.342133 25130 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:37.342778 25130 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:37.344436 24854 ts_manager.cc:194] Registered new tserver with Master: 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:37.344772 23796 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011837281s
I20240426 13:18:37.345616 24854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39858
I20240426 13:18:37.347024 23796 test_util.cc:268] Using random seed: -1110355777
I20240426 13:18:37.367458 24854 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39872:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240426 13:18:37.373580 24854 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240426 13:18:37.577472 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.579034 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.579386 24944 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.580775 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.581259 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.582528 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.585220 24943 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.586447 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.589200 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.590479 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.590533 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.591826 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.593201 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.594519 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.595153 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.596433 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.597169 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.598407 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.601140 24939 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.602380 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.605196 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.606555 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.609135 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.610384 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.613425 25093 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.615087 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.617429 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.618050 25145 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:37.618759 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.621213 25088 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.622548 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.625223 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.625478 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.626467 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.626544 25145 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.627086 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.628460 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.629717 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.634732 25095 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.641392 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.641959 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.642685 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.637342 25091 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.646632 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.639305 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.648916 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.648602 25094 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.651011 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.661221 25095 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.662990 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.671178 25094 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.673141 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.684391 25014 tablet_service.cc:1467] Processing CreateTablet for tablet f49912f064d045d8a1d8be2c08b97d44 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240426 13:18:37.685154 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 53343363b25040c29f4c212b440dc2cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240426 13:18:37.685724 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.686363 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.687680 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 59632e528afa44ca98a626355e1e4660 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240426 13:18:37.688956 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.692350 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 718dfae59b1148c2b6cc522821ea4ea4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240426 13:18:37.693193 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1db66b7c1043e1bd43d7254a4dec62 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240426 13:18:37.693643 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718dfae59b1148c2b6cc522821ea4ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.694416 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1db66b7c1043e1bd43d7254a4dec62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.697156 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 08f19147e9214114a8749e2a8196bacd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240426 13:18:37.698418 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f19147e9214114a8749e2a8196bacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.701140 25019 tablet_service.cc:1467] Processing CreateTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240426 13:18:37.702380 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.697155 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 1708857f1bab4329a3f12081a3c1d669 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240426 13:18:37.705394 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708857f1bab4329a3f12081a3c1d669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.705107 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240426 13:18:37.713137 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 499552c0877e46c5af41389ff1be3152 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240426 13:18:37.725768 25094 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:37.729112 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e6a71a0c24a3aa4f1348fd2ae9fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.734915 24942 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.751345 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499552c0877e46c5af41389ff1be3152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.751631 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.752288 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.736433 25092 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.740459 24940 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.719669 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.755167 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.755297 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.746276 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.756772 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.757977 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.748661 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 11fad756437243d2959f5998d6bf8309 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240426 13:18:37.750306 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.764757 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.768314 24944 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.769526 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.775228 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.804873 24941 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.805512 24946 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:37.806028 25145 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:37.806178 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.806805 25145 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.189s user 0.019s sys 0.020s
I20240426 13:18:37.808923 25016 tablet_service.cc:1467] Processing CreateTablet for tablet a6a69cdcf2ee4768a20888dd974c6ba7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240426 13:18:37.809787 24943 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.811029 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.813645 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.815013 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.815280 25148 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:37.815414 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.810181 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a69cdcf2ee4768a20888dd974c6ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.822288 25091 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.823637 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.830816 25012 tablet_service.cc:1467] Processing CreateTablet for tablet a73f20bc0b224284bc0bf8eed62401d6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240426 13:18:37.832160 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73f20bc0b224284bc0bf8eed62401d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.853003 25093 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.855770 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.857807 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:37.853677 25145 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:37.858708 25145 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:37.859046 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.867372 25020 tablet_service.cc:1467] Processing CreateTablet for tablet a2212db2dbcf486888aefb7dd0174287 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240426 13:18:37.868748 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.869307 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.870513 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.875072 25087 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:37.876885 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.876590 25088 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.877725 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.878090 25148 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.878361 25147 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:37.880193 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:18:37.881896 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.880295 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.892767 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.897047 25017 tablet_service.cc:1467] Processing CreateTablet for tablet aca63e93aec8413b83ec63e853c8f403 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240426 13:18:37.898214 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca63e93aec8413b83ec63e853c8f403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.888609 25015 tablet_service.cc:1467] Processing CreateTablet for tablet b9910a5336aa4e3cab11dca4a556fabf (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240426 13:18:37.902819 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.859057 25145 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:37.904639 25145 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:37.908771 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe8f3015c01480fb67b39d3f5a947c7 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240426 13:18:37.909963 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe8f3015c01480fb67b39d3f5a947c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.915614 25147 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:37.917304 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.918517 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.918411 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:37.923496 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:37.926959 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8fda82f95348219809212d99d6704e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240426 13:18:37.906509 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.928849 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.930061 25013 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240426 13:18:37.931424 24938 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.932668 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.933267 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:37.934392 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.936722 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.938972 24946 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.940659 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.943996 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.942648 24943 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:37.961514 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.945870 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 551154b9c3494e4da288affc4b88c361 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240426 13:18:37.936595 25016 tablet_service.cc:1467] Processing CreateTablet for tablet d066b73b2e464d80b7b667e05602d879 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240426 13:18:37.970690 25130 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:37.973150 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:37.957049 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8fda82f95348219809212d99d6704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.002503 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:18:37.978174 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:38.004139 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.982647 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:37.983345 25020 tablet_service.cc:1467] Processing CreateTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240426 13:18:37.986541 25093 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.007699 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.008579 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.987274 25091 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:37.990264 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:37.953018 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 786abf3f43724ad5b959e497fc776d1d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240426 13:18:38.017274 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.991297 25086 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:37.983907 25145 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.177s user 0.003s sys 0.009s
I20240426 13:18:38.018601 25145 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:37.995186 25149 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.019124 25149 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.019580 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.024718 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.998539 25148 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:37.891194 25021 tablet_service.cc:1467] Processing CreateTablet for tablet f7408112bace482693c999f678aa9d73 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240426 13:18:38.004623 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4cb6239dd94f7693fdaacdc8ae8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:37.981627 24944 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240426 13:18:38.029587 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7408112bace482693c999f678aa9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.030187 25148 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.215s user 0.005s sys 0.014s
I20240426 13:18:38.030505 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.034312 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.040359 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551154b9c3494e4da288affc4b88c361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.044459 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:18:38.044231 24840 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 54776.8us
I20240426 13:18:38.050207 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:38.051501 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.052965 25149 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.057695 25145 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.068341 25015 tablet_service.cc:1467] Processing CreateTablet for tablet f533a7c13bae41539cdc0a425b282e53 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240426 13:18:38.069777 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f533a7c13bae41539cdc0a425b282e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.073909 25147 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.051357 25148 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.076084 25148 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.076387 25148 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.077122 25148 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:38.083885 25131 tablet.cc:2369] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240426 13:18:38.085844 25088 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.087085 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.090071 25147 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.212s user 0.005s sys 0.009s
I20240426 13:18:38.090942 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:38.100340 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.101540 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 3014c46944bc4cbb82624ac909221634 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
W20240426 13:18:38.102592 25069 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.103657 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.109357 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.112370 24941 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.113957 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.117508 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:38.118702 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.127431 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.128734 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.133718 25147 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.144456 25147 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.144779 25147 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.145462 25147 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.148523 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 4680c139c46f44b49096bfd65f033a32 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240426 13:18:38.150168 25087 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.149289 25090 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.158952 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.164444 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.164609 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.168125 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.168211 24981 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:38.181056 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240426 13:18:38.182287 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.189365 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.191462 25145 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.191316 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.191548 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:38.192519 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.192772 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.193048 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240426 13:18:38.194216 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.195780 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 256415e42b6c444387115ab766a5a16c (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240426 13:18:38.197000 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.197291 25013 tablet_service.cc:1467] Processing CreateTablet for tablet f4076b67a4fe4b3297158d094e10de08 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240426 13:18:38.198519 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.200145 25145 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.174s user 0.013s sys 0.000s
I20240426 13:18:38.201457 25056 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:38.159952 25093 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.204003 25145 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.206416 25091 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.228255 25145 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.207374 25148 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.177s user 0.009s sys 0.012s
I20240426 13:18:38.222730 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.228703 25145 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.229168 25147 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.139s user 0.003s sys 0.019s
I20240426 13:18:38.229351 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217453 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.213820 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.229674 25145 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.230813 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.232126 25145 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s user 0.003s sys 0.004s
I20240426 13:18:38.233109 25145 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.236304 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217965 25086 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.224579 25090 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.237174 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.237821 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.217831 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.239202 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.241080 25145 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.225517 25088 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.244215 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.215530 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.245808 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.246529 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.229369 25148 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.245366 25092 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.256443 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.245357 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.213295 25019 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:38.264731 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.268968 25147 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:38.279460 25148 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.286968 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.287640 25090 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.288264 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.281742 25145 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.285348 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.289825 25145 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.031s
I20240426 13:18:38.290482 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.292508 25145 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.293277 25147 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.293227 25145 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.286551 25093 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.300252 25145 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.300889 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.300957 25145 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.301950 25091 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:18:38.303153 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.303220 25145 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240426 13:18:38.303400 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.304415 25145 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.332094 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240426 13:18:38.335355 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.163596 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 6324309bd7cf40778c508e740dcf979d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240426 13:18:38.350076 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6324309bd7cf40778c508e740dcf979d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.179656 24944 tablet_service.cc:1467] Processing CreateTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240426 13:18:38.357049 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.376466 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 009697c9dea4421d868c467f0cd6c079 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240426 13:18:38.377954 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009697c9dea4421d868c467f0cd6c079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.379899 25019 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.381626 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.383262 25015 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.385766 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.384589 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.355283 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240426 13:18:38.392932 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.395599 24943 tablet_service.cc:1467] Processing CreateTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240426 13:18:38.393014 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.408063 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd5654ae4294fcc8482e45f83032773 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240426 13:18:38.409291 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cd5654ae4294fcc8482e45f83032773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.412300 25019 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.414443 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.417305 25017 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.418910 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.419075 25020 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.421002 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:38.426424 25070 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.427179 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.421013 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.433051 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.433512 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.435139 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.435894 25090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.441484 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.295799 25092 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.461340 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.466588 25013 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.468446 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.473865 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.480684 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.467995 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.495630 25148 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.496421 25148 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.267s user 0.012s sys 0.008s
I20240426 13:18:38.497617 25086 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.499004 25086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.502832 25149 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.502971 25155 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.513041 25155 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.517927 25019 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.519392 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.520320 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.523912 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.528825 25155 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.532027 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.536592 25157 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.539347 25157 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.543363 24942 tablet_service.cc:1467] Processing CreateTablet for tablet c896b58b7766401eac0a7f69574827f1 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240426 13:18:38.499835 25148 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.563165 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.563630 25149 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.564766 25149 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.566833 25149 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.567756 25157 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.568092 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.568682 24956 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:38.570217 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.572024 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:18:38.572959 25070 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.573477 25015 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.573762 25149 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:38.579622 25145 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.581264 25018 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.587352 25012 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.570097 25017 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.556891 25148 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.588941 25149 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:38.570058 25087 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.596726 25149 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.597409 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.604624 24945 tablet_service.cc:1467] Processing CreateTablet for tablet f04bd8996e6349879e25fca4e52dbda9 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240426 13:18:38.605875 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04bd8996e6349879e25fca4e52dbda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.610054 25095 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.611248 25095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.612457 24940 tablet_service.cc:1467] Processing CreateTablet for tablet 6277899576264e68aa43514e5b5c676e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240426 13:18:38.613544 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.619005 25147 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.619611 25147 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.351s user 0.022s sys 0.008s
I20240426 13:18:38.620436 25017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.620509 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.620625 25149 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.621606 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.623450 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:38.624037 24956 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.624316 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.593691 25148 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.635706 24956 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:38.638993 25093 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.649586 25013 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.657608 25093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.654426 25015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.663815 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.662847 24943 tablet_service.cc:1467] Processing CreateTablet for tablet 9108945c08ba4b2287c1e238b402722f (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240426 13:18:38.669503 25149 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.677589 25166 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.667884 24938 tablet_service.cc:1467] Processing CreateTablet for tablet eb99666c521d49ada45165233068daf3 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240426 13:18:38.678628 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:38.679461 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.669657 24941 tablet_service.cc:1467] Processing CreateTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240426 13:18:38.683959 25014 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.663935 25089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 47) took 1022 ms. Trace:
I20240426 13:18:38.684181 25148 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.686050 25092 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 52) took 1031 ms. Trace:
I20240426 13:18:38.688015 25090 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 51) took 1036 ms. Trace:
I20240426 13:18:38.688159 25088 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 45) took 1014 ms. Trace:
I20240426 13:18:38.696297 25088 rpcz_store.cc:276] 0426 13:18:37.640996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.224681 (+583685us) service_pool.cc:225] Handling call
0426 13:18:38.244300 (+ 19619us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.245415 (+ 1115us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.271742 (+ 26327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.616302 (+344560us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.655355 (+ 39053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.689150 25020 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.702423 25147 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.703464 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.703419 25147 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.703953 25147 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.705761 25090 rpcz_store.cc:276] 0426 13:18:37.651962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.286829 (+634867us) service_pool.cc:225] Handling call
0426 13:18:38.288445 (+ 1616us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.288542 (+ 97us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.587085 (+298543us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.687524 (+100439us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.687964 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.694027 25166 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.699427 25148 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.200s user 0.007s sys 0.000s
W20240426 13:18:38.701098 24840 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68490.9us
I20240426 13:18:38.670660 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:38.715652 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.708540 25149 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:38.724191 25019 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20240426 13:18:38.730082 25070 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.731668 25019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.732003 25016 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.732288 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.733325 25016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.733606 25091 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 53) took 1076 ms. Trace:
W20240426 13:18:38.733882 24996 leader_election.cc:341] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.738577 25148 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.694818 25092 rpcz_store.cc:276] 0426 13:18:37.654448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.294941 (+640493us) service_pool.cc:225] Handling call
0426 13:18:38.460631 (+165690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.460754 (+ 123us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.572134 (+111380us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.685629 (+113495us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.686000 (+ 371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.693457 25089 rpcz_store.cc:276] 0426 13:18:37.641629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.431506 (+789877us) service_pool.cc:225] Handling call
0426 13:18:38.434172 (+ 2666us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.434340 (+ 168us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.628846 (+194506us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.663472 (+ 34626us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.663865 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.745680 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.748142 25166 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:38.748077 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:38.749526 25070 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:38.748081 24995 leader_election.cc:341] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.750317 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:38.751406 25155 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.752310 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.723881 25147 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.753031 25157 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:38.753551 25069 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.753438 25157 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.753968 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.667615 24946 tablet_service.cc:1467] Processing CreateTablet for tablet bee343a4cf024dd09c3714d8cc509694 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240426 13:18:38.739169 25087 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 56) took 1074 ms. Trace:
I20240426 13:18:38.739670 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9108945c08ba4b2287c1e238b402722f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.759843 25157 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.770581 25093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 58) took 1099 ms. Trace:
I20240426 13:18:38.770362 25021 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.788829 25021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.761605 25091 rpcz_store.cc:276] 0426 13:18:37.656907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.301103 (+644196us) service_pool.cc:225] Handling call
0426 13:18:38.302667 (+ 1564us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.302781 (+ 114us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.583905 (+281124us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.732922 (+149017us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.733383 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.778270 25093 rpcz_store.cc:276] 0426 13:18:37.671196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.638075 (+966879us) service_pool.cc:225] Handling call
0426 13:18:38.657133 (+ 19058us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.657246 (+ 113us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.697004 (+ 39758us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.770172 (+ 73168us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.770527 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.763131 25166 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.762547 25086 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 55) took 1100 ms. Trace:
I20240426 13:18:38.794020 25149 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:38.797530 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.798414 25148 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.798441 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.801273 24939 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 27) took 1082 ms. Trace:
I20240426 13:18:38.802131 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.802588 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 32) took 1010 ms. Trace:
I20240426 13:18:38.802444 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 9911e0422347463c8781b2574c023744 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240426 13:18:38.804675 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6f5c5283464089ad1bc18c13b427cc (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240426 13:18:38.809494 25104 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:38.807394 25086 rpcz_store.cc:276] 0426 13:18:37.662332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.496395 (+834063us) service_pool.cc:225] Handling call
0426 13:18:38.498500 (+ 2105us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.498607 (+ 107us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.642656 (+144049us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.739661 (+ 97005us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.761622 (+ 21961us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.810154 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.807978 24939 rpcz_store.cc:276] 0426 13:18:37.718898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.045613 (+326715us) service_pool.cc:225] Handling call
0426 13:18:38.051051 (+ 5438us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.051147 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.141371 (+ 90224us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.699185 (+557814us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.801106 (+101921us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7040}
I20240426 13:18:38.811470 24944 rpcz_store.cc:276] 0426 13:18:37.794713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.178830 (+384117us) service_pool.cc:225] Handling call
0426 13:18:38.356260 (+177430us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.356412 (+ 152us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.564843 (+208431us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.730058 (+165215us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.802529 (+ 72471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.812446 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee343a4cf024dd09c3714d8cc509694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.813334 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6f5c5283464089ad1bc18c13b427cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.813652 25145 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.813304 24944 tablet_service.cc:1467] Processing CreateTablet for tablet 242110e36ca34176b9d522087d440de6 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240426 13:18:38.814126 25145 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.510s user 0.009s sys 0.007s
I20240426 13:18:38.814575 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242110e36ca34176b9d522087d440de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.817559 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:38.818099 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:38.818329 25157 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:38.818686 25157 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:38.819008 25157 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:38.819835 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9911e0422347463c8781b2574c023744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.820586 24939 tablet_service.cc:1467] Processing CreateTablet for tablet 81d7dd33a3494a2d94627156d2fc1615 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240426 13:18:38.821913 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81d7dd33a3494a2d94627156d2fc1615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.808704 25017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 49) took 1008 ms. Trace:
W20240426 13:18:38.824800 24921 leader_election.cc:341] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.825289 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:38.826303 24921 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.824985 25017 rpcz_store.cc:276] 0426 13:18:37.799904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.569251 (+769347us) service_pool.cc:225] Handling call
0426 13:18:38.619407 (+ 50156us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.619652 (+ 245us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.736743 (+117091us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.760376 (+ 23633us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.808021 (+ 47645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.761816 25087 rpcz_store.cc:276] 0426 13:18:37.665197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.568975 (+903778us) service_pool.cc:225] Handling call
0426 13:18:38.596950 (+ 27975us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.597058 (+ 108us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.610318 (+ 13260us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.738672 (+128354us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.739106 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.828996 25157 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.819490 25015 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 50) took 1003 ms. Trace:
I20240426 13:18:38.830808 25157 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.831357 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb99666c521d49ada45165233068daf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.830817 25015 rpcz_store.cc:276] 0426 13:18:37.815913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.572571 (+756658us) service_pool.cc:225] Handling call
0426 13:18:38.595550 (+ 22979us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.627194 (+ 31644us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.697695 (+ 70501us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.819020 (+121325us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.819437 (+ 417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1667840}
I20240426 13:18:38.838073 25018 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.838618 25013 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 52) took 1019 ms. Trace:
I20240426 13:18:38.839215 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.838984 25013 rpcz_store.cc:276] 0426 13:18:37.818927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.648650 (+829723us) service_pool.cc:225] Handling call
0426 13:18:38.658044 (+ 9394us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.664297 (+ 6253us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.801616 (+137319us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.838188 (+ 36572us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.838567 (+ 379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.840343 25149 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:38.842885 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 51) took 1024 ms. Trace:
I20240426 13:18:38.843271 25012 rpcz_store.cc:276] 0426 13:18:37.818517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.586445 (+767928us) service_pool.cc:225] Handling call
0426 13:18:38.627182 (+ 40737us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.630232 (+ 3050us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.727521 (+ 97289us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.842333 (+114812us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.842814 (+ 481us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4608}
I20240426 13:18:38.831292 25157 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.847242 25095 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:58852 (request call id 57) took 1179 ms. Trace:
I20240426 13:18:38.847589 25095 rpcz_store.cc:276] 0426 13:18:37.667940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.609099 (+941159us) service_pool.cc:225] Handling call
0426 13:18:38.610789 (+ 1690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.610890 (+ 101us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.767481 (+156591us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.846786 (+ 79305us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.847188 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.848778 25145 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.849534 25145 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.849836 25145 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.850534 25145 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.853025 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:38.857121 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:38.843807 24940 tablet_service.cc:1467] Processing CreateTablet for tablet ece98ef0f07f4184bedf2cb96b628f20 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240426 13:18:38.857630 25105 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:38.857694 24921 leader_election.cc:341] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.868252 25148 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.868983 25148 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.146s user 0.008s sys 0.003s
I20240426 13:18:38.868417 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:38.869544 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:38.870934 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:38.872145 24940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece98ef0f07f4184bedf2cb96b628f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.872624 25170 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:38.873946 25170 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:38.874706 25170 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.875326 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 48) took 1016 ms. Trace:
I20240426 13:18:38.875711 24937 rpcz_store.cc:276] 0426 13:18:37.859193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.803347 (+944154us) service_pool.cc:225] Handling call
0426 13:18:38.812912 (+ 9565us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.812998 (+ 86us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.846005 (+ 33007us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.874918 (+ 28913us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.875260 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.877697 25145 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.063s user 0.005s sys 0.003s
I20240426 13:18:38.882521 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240426 13:18:38.883675 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.883932 24941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 42) took 1026 ms. Trace:
I20240426 13:18:38.884352 24941 rpcz_store.cc:276] 0426 13:18:37.857225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.643845 (+786620us) service_pool.cc:225] Handling call
0426 13:18:38.801616 (+157771us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.801728 (+ 112us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.837918 (+ 36190us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.883453 (+ 45535us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.883813 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.886355 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 47) took 1027 ms. Trace:
I20240426 13:18:38.886881 24944 rpcz_store.cc:276] 0426 13:18:37.858890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.812462 (+953572us) service_pool.cc:225] Handling call
0426 13:18:38.814152 (+ 1690us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.814236 (+ 84us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.849563 (+ 35327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.885915 (+ 36352us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.886303 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.888217 24943 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 43) took 1030 ms. Trace:
I20240426 13:18:38.888633 24943 rpcz_store.cc:276] 0426 13:18:37.857553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.661992 (+804439us) service_pool.cc:225] Handling call
0426 13:18:38.693736 (+ 31744us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.714567 (+ 20831us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.838894 (+124327us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.887738 (+ 48844us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.888162 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.889163 24944 tablet_service.cc:1467] Processing CreateTablet for tablet aca59ad77ad642c48a7e7eed1635089d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240426 13:18:38.881547 24945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 39) took 1025 ms. Trace:
I20240426 13:18:38.890983 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca59ad77ad642c48a7e7eed1635089d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.890741 24945 rpcz_store.cc:276] 0426 13:18:37.856048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.571894 (+715846us) service_pool.cc:225] Handling call
0426 13:18:38.605407 (+ 33513us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.605514 (+ 107us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.776648 (+171134us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.861064 (+ 84416us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.881002 (+ 19938us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.879767 25019 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 53) took 1060 ms. Trace:
I20240426 13:18:38.895174 25147 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.275s user 0.014s sys 0.018s
I20240426 13:18:38.894203 25019 rpcz_store.cc:276] 0426 13:18:37.856142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.658082 (+801940us) service_pool.cc:225] Handling call
0426 13:18:38.731194 (+ 73112us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.731288 (+ 94us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.811123 (+ 79835us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.860538 (+ 49415us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.879219 (+ 18681us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6823}
I20240426 13:18:38.898324 25145 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:38.900758 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 680b3a9e21d44d71987bc91ce797f547 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240426 13:18:38.902603 25148 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.903857 25148 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.910308 25148 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.911708 25147 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:38.911263 25148 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.912457 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 50) took 1052 ms. Trace:
I20240426 13:18:38.913219 25145 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.914101 24946 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 44) took 1056 ms. Trace:
I20240426 13:18:38.914505 24946 rpcz_store.cc:276] 0426 13:18:37.857864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.666615 (+808751us) service_pool.cc:225] Handling call
0426 13:18:38.804131 (+137516us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.804262 (+ 131us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.842703 (+ 38441us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.902070 (+ 59367us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.913781 (+ 11711us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.914046 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1014,"spinlock_wait_cycles":15265152}
I20240426 13:18:38.906599 24942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 45) took 1048 ms. Trace:
I20240426 13:18:38.908224 24943 tablet_service.cc:1467] Processing CreateTablet for tablet dacba3e2733347ecb1304d307310c944 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240426 13:18:38.908061 24854 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:38.918083 24942 rpcz_store.cc:276] 0426 13:18:37.858177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.801473 (+943296us) service_pool.cc:225] Handling call
0426 13:18:38.819414 (+ 17941us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.819496 (+ 82us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.854281 (+ 34785us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.906158 (+ 51877us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.906548 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.909647 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680b3a9e21d44d71987bc91ce797f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.886359 24941 tablet_service.cc:1467] Processing CreateTablet for tablet edc419d5057b449c8d0888a41017ce2e (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240426 13:18:38.907943 24938 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 41) took 1051 ms. Trace:
I20240426 13:18:38.913285 24940 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 49) took 1053 ms. Trace:
I20240426 13:18:38.920069 24942 tablet_service.cc:1467] Processing CreateTablet for tablet 99a876ca7b6e4082b58359403a47730d (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240426 13:18:38.917714 25148 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.048s user 0.001s sys 0.004s
I20240426 13:18:38.920842 24938 rpcz_store.cc:276] 0426 13:18:37.856652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.611570 (+754918us) service_pool.cc:225] Handling call
0426 13:18:38.800720 (+189150us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b539c 00007fda8d3b4f5b 00007fda962a76d9 00007fda951e4270 00007fda952c6651 00007fda952a3dc3 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2 00007fda8e853af1
0426 13:18:38.805314 (+ 4594us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.805488 (+ 174us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.880197 (+ 74709us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.907506 (+ 27309us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.907890 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19498880}
I20240426 13:18:38.912845 24937 rpcz_store.cc:276] 0426 13:18:37.859782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.880621 (+1020839us) service_pool.cc:225] Handling call
0426 13:18:38.883276 (+ 2655us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.883358 (+ 82us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.890677 (+ 7319us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.911831 (+ 21154us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.912386 (+ 555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.923012 24943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacba3e2733347ecb1304d307310c944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.923900 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.925019 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 79bb183792fd4ae08dd4564ec82efc36 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240426 13:18:38.926208 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bb183792fd4ae08dd4564ec82efc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.926515 25018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 58) took 1068 ms. Trace:
I20240426 13:18:38.921234 24940 rpcz_store.cc:276] 0426 13:18:37.859505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.842871 (+983366us) service_pool.cc:225] Handling call
0426 13:18:38.858222 (+ 15351us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.858802 (+ 580us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.885829 (+ 27027us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.891729 (+ 5900us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.912954 (+ 21225us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.913220 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22257408}
I20240426 13:18:38.926895 25018 rpcz_store.cc:276] 0426 13:18:37.857693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.775706 (+918013us) service_pool.cc:225] Handling call
0426 13:18:38.838768 (+ 63062us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.838870 (+ 102us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.879927 (+ 41057us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.922709 (+ 42782us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.926460 (+ 3751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.923012 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 52) took 1018 ms. Trace:
I20240426 13:18:38.929945 24944 rpcz_store.cc:276] 0426 13:18:37.904693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.888330 (+983637us) service_pool.cc:225] Handling call
0426 13:18:38.890557 (+ 2227us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.890646 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.906748 (+ 16102us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.922492 (+ 15744us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.922938 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.932478 24944 tablet_service.cc:1467] Processing CreateTablet for tablet e915e90a1a6847899575125ef7e68f2a (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240426 13:18:38.933271 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.925386 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 (DEFAULT_TABLE table=test-workload [id=a0cdcee3c79d424793c3372b06b3b129]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240426 13:18:38.936635 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.938041 24939 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 46) took 1079 ms. Trace:
I20240426 13:18:38.938405 24939 rpcz_store.cc:276] 0426 13:18:37.858525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.811587 (+953062us) service_pool.cc:225] Handling call
0426 13:18:38.821372 (+ 9785us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.821592 (+ 220us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.863982 (+ 42390us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.902357 (+ 38375us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.936623 (+ 34266us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.937987 (+ 1364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30870400}
I20240426 13:18:38.946869 24937 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 56) took 1041 ms. Trace:
I20240426 13:18:38.935052 25021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51352 (request call id 57) took 1077 ms. Trace:
I20240426 13:18:38.948401 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:38.948212 25021 rpcz_store.cc:276] 0426 13:18:37.857411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.769160 (+911749us) service_pool.cc:225] Handling call
0426 13:18:38.788183 (+ 19023us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.788312 (+ 129us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.867791 (+ 79479us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.934564 (+ 66773us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.934997 (+ 433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.958647 25145 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:38.959172 25145 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.013s
I20240426 13:18:38.960166 24943 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 53) took 1055 ms. Trace:
I20240426 13:18:38.935849 25148 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:38.965799 24943 rpcz_store.cc:276] 0426 13:18:37.904983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.889625 (+984642us) service_pool.cc:225] Handling call
0426 13:18:38.918253 (+ 28628us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.918348 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.933646 (+ 15298us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.959647 (+ 26001us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.960091 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.935398 25147 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.968609 24938 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 57) took 1062 ms. Trace:
I20240426 13:18:38.969004 24938 rpcz_store.cc:276] 0426 13:18:37.906146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.923593 (+1017447us) service_pool.cc:225] Handling call
0426 13:18:38.936639 (+ 13046us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.936747 (+ 108us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.958036 (+ 21289us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.968121 (+ 10085us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.968542 (+ 421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.970541 24945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 54) took 1065 ms. Trace:
I20240426 13:18:38.970975 24945 rpcz_store.cc:276] 0426 13:18:37.905271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.892851 (+987580us) service_pool.cc:225] Handling call
0426 13:18:38.909107 (+ 16256us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952c6651 00007fda952a3dc3 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2 00007fda8e853af1
0426 13:18:38.909220 (+ 113us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.909315 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.945091 (+ 35776us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.970033 (+ 24942us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.970468 (+ 435us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10339712}
I20240426 13:18:38.972115 25148 tablet_bootstrap.cc:654] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.947249 24937 rpcz_store.cc:276] 0426 13:18:37.905858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.923560 (+1017702us) service_pool.cc:225] Handling call
0426 13:18:38.925808 (+ 2248us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.925894 (+ 86us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.938871 (+ 12977us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.946392 (+ 7521us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.946810 (+ 418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.961813 25145 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.965438 25155 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.973376 25155 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.975497 25155 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.976500 25145 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.976795 25145 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.977093 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:38.977526 25145 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.960458 24944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 58) took 1054 ms. Trace:
W20240426 13:18:38.978317 24995 leader_election.cc:341] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.978482 24944 rpcz_store.cc:276] 0426 13:18:37.906444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.931403 (+1024959us) service_pool.cc:225] Handling call
0426 13:18:38.933165 (+ 1762us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.933249 (+ 84us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.953818 (+ 20569us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.960001 (+ 6183us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.960406 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.980715 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:38.981206 25145 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s user 0.002s sys 0.006s
I20240426 13:18:38.982242 25145 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:38.982257 24996 leader_election.cc:341] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:38.982887 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:38.984021 25155 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:38.986543 25147 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:38.987112 25147 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.001s
I20240426 13:18:38.987656 24941 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 51) took 1083 ms. Trace:
I20240426 13:18:38.988101 24941 rpcz_store.cc:276] 0426 13:18:37.904323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.885497 (+981174us) service_pool.cc:225] Handling call
0426 13:18:38.920715 (+ 35218us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.920821 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.961513 (+ 40692us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.987050 (+ 25537us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.987596 (+ 546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:18:38.990537 24942 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41280 (request call id 55) took 1085 ms. Trace:
I20240426 13:18:38.990562 25147 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.991753 25147 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.992280 25147 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.993153 25147 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.991086 24942 rpcz_store.cc:276] 0426 13:18:37.905572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:18:38.919184 (+1013612us) service_pool.cc:225] Handling call
0426 13:18:38.921842 (+ 2658us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0426 13:18:38.922009 (+ 167us) ts_tablet_manager.cc:655] Creating new metadata...
0426 13:18:38.954854 (+ 32845us) tablet_metadata.cc:641] Metadata flushed
0426 13:18:38.983809 (+ 28955us) tablet_replica.cc:205] Creating consensus instance
0426 13:18:38.990255 (+ 6446us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b56a2 00007fda962d0449 00007fda951e4118 00007fda952a7bf3 00007fda952a3a0f 00007fda952a41c1 00007fda952554d9 00007fda8e853c34 00007fda8e853bc2
0426 13:18:38.990484 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5518080}
I20240426 13:18:38.995005 25148 tablet_bootstrap.cc:492] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:38.995123 25145 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:38.995541 25148 ts_tablet_manager.cc:1367] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.012s
I20240426 13:18:38.995971 25155 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:38.996450 25155 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.998348 25155 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:38.998412 25148 raft_consensus.cc:348] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:38.999351 25148 raft_consensus.cc:374] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:38.999666 25148 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:38.999890 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.000552 25148 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:39.001170 24996 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.002254 25147 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s user 0.009s sys 0.001s
I20240426 13:18:39.002501 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.003217 25147 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
W20240426 13:18:39.004426 24995 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.004999 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.005849 25148 ts_tablet_manager.cc:1398] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240426 13:18:39.006906 25148 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.009493 25147 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.013036 25148 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.013547 25155 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.021522 25147 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.022022 25147 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s
I20240426 13:18:39.023977 25148 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.024493 25148 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.002s
I20240426 13:18:39.024854 25147 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.025617 25147 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.025907 25147 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.026670 25147 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.027623 25148 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.028415 25148 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.028787 25148 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.029227 25147 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240426 13:18:39.029678 25148 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.030246 25147 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.031438 25145 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.031921 25145 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.017s
I20240426 13:18:39.032493 25148 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 13:18:39.033533 25148 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.034509 25145 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.035243 25145 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.035534 25145 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.036674 25147 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.036219 25145 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.038551 25145 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:39.039611 25145 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.040655 25148 tablet_bootstrap.cc:654] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.045409 25145 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.057101 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:39.059366 25147 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.059463 25148 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.059885 25147 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.009s
I20240426 13:18:39.060199 25148 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.014s
I20240426 13:18:39.062747 25147 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.062901 25148 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.063585 25147 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.063690 25148 raft_consensus.cc:374] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.063978 25147 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.064024 25148 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.064951 25145 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.065443 25145 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.001s
I20240426 13:18:39.064834 25147 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.064925 25148 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.067078 25147 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:18:39.068457 25147 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.067999 25145 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.068886 25145 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.069166 25145 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.069871 25145 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.070523 25148 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240426 13:18:39.071511 25148 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.072081 25145 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20240426 13:18:39.072903 25145 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.074620 25147 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.078373 25145 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.078459 25155 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.079080 25155 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.083289 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.083989 24956 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.085317 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.086015 25147 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.086454 25147 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240426 13:18:39.086736 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.085907 25105 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.090493 25145 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.091064 25145 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20240426 13:18:39.087991 25155 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.091876 25155 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.092651 25155 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.092967 25155 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.094166 25148 tablet_bootstrap.cc:654] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.094157 25145 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.095037 25145 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.095278 25145 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.095932 25145 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.098698 25145 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240426 13:18:39.099750 25145 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.104473 25155 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.106462 25145 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.106622 25147 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.107363 25147 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.107640 25147 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.108325 25147 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.109539 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.110065 24956 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.110132 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.110466 25147 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.024s user 0.001s sys 0.005s
I20240426 13:18:39.110736 25105 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.113968 25147 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.117724 25105 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.118468 25145 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.118917 25145 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.002s
I20240426 13:18:39.118777 24996 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.119697 25170 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.119819 25190 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.118815 24956 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.120213 25170 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.122236 25145 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.123148 25145 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.123497 25145 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.124341 25145 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.125242 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.125859 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.126641 25147 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.127193 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.120995 25155 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.127979 25157 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.128324 25157 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.128641 25157 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.130584 25145 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240426 13:18:39.131491 25145 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.131531 25190 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:39.132632 25170 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.138370 25147 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.133075 25190 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.138818 25147 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.004s
I20240426 13:18:39.134474 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.141188 25147 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.141901 25147 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.138592 25145 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.142233 24921 leader_election.cc:341] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.142213 25147 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.122407 25148 tablet_bootstrap.cc:492] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.143832 25148 ts_tablet_manager.cc:1367] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.012s
I20240426 13:18:39.143949 25147 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.145356 25169 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.145766 25169 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.147686 25169 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.149708 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.150998 25070 leader_election.cc:341] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.142457 25157 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.153137 25147 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240426 13:18:39.153836 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.154332 25031 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.160684 25147 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.161283 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.162408 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.163288 25170 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.167680 25147 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.169404 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.169744 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.170105 25157 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:39.172283 24921 leader_election.cc:341] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.173187 25145 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.173650 25145 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.017s
W20240426 13:18:39.174160 25069 leader_election.cc:341] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.174660 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.176189 25169 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.176386 25145 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.177345 25145 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.177677 25145 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.178462 25145 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.179260 25170 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.180539 25145 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240426 13:18:39.180109 25170 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.181711 25145 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.182786 25147 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.183246 25147 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20240426 13:18:39.185091 25197 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.185572 25197 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.185971 25147 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.186764 25147 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.187091 25147 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.187391 25145 tablet_bootstrap.cc:654] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.187878 25147 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.188385 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.190770 25147 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20240426 13:18:39.190721 24921 leader_election.cc:341] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.188781 25148 raft_consensus.cc:348] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.192148 25148 raft_consensus.cc:374] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.192550 25147 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.192528 25148 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.194869 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.195369 25197 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.195506 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.196743 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.196537 25148 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.198446 25147 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.198544 25197 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.199040 25197 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.199505 25197 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.206655 24852 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.208439 25145 tablet_bootstrap.cc:492] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.208865 25145 ts_tablet_manager.cc:1367] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.015s
I20240426 13:18:39.208976 25147 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.209354 25147 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240426 13:18:39.211633 25147 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.212231 25147 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.211843 25145 raft_consensus.cc:348] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.212495 25147 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.212601 25145 raft_consensus.cc:374] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.212888 25145 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.213148 25147 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.215050 25147 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240426 13:18:39.215451 25148 ts_tablet_manager.cc:1398] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.071s user 0.005s sys 0.002s
I20240426 13:18:39.215860 25147 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.216468 25148 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.215847 24851 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.213485 25145 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.220806 25145 ts_tablet_manager.cc:1398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240426 13:18:39.221727 25147 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.221968 25145 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.225143 25197 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.227339 25197 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.227505 25145 tablet_bootstrap.cc:654] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.227862 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.229904 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.230405 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.231024 25148 tablet_bootstrap.cc:654] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.234544 24921 leader_election.cc:341] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.235648 25190 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.236238 25190 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:18:39.238678 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.240095 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.240316 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.241425 24956 raft_consensus.cc:2379] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:39.241255 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.246407 25190 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:39.247105 25197 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.247915 25197 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.250922 25197 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.250443 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.292766 25148 tablet_bootstrap.cc:492] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.289958 25145 tablet_bootstrap.cc:492] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.291677 25149 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240426 13:18:39.303798 25145 ts_tablet_manager.cc:1367] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.082s user 0.023s sys 0.016s
I20240426 13:18:39.303721 25148 ts_tablet_manager.cc:1367] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.087s user 0.018s sys 0.024s
I20240426 13:18:39.304236 25147 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.304695 25147 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.012s
I20240426 13:18:39.306543 25145 raft_consensus.cc:348] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.306963 25147 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.256834 25190 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.307652 25147 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.306850 25148 raft_consensus.cc:348] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.307816 25190 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.307353 25145 raft_consensus.cc:374] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.290985 25169 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.308665 25031 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.310665 25148 raft_consensus.cc:374] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.311134 25148 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.311933 25148 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.312285 25169 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.313620 25148 ts_tablet_manager.cc:1398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240426 13:18:39.307947 25147 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.315210 25147 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.315562 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.317772 25148 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.317582 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.318046 25169 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:39.318832 25070 leader_election.cc:341] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.318503 25145 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.319857 25145 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.321429 24955 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:39.321911 25145 ts_tablet_manager.cc:1398] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240426 13:18:39.322484 25169 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.324317 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.324859 25148 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.325687 25147 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240426 13:18:39.325947 25169 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.326294 25169 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.326612 25147 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.326822 25169 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.326639 25190 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.328858 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.329188 25190 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.329558 25105 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:39.329587 25190 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.332399 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.332957 25147 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.332927 25105 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.334062 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.334889 25193 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.335269 25193 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.335558 25193 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:39.335845 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91252us
I20240426 13:18:39.337294 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.343394 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.344094 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:39.344612 25145 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.346203 25155 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.347292 25193 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.349409 25147 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.349814 25147 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20240426 13:18:39.350580 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.351176 25105 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.351770 25148 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.352319 25193 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.352252 25148 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20240426 13:18:39.353039 25170 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.353531 25170 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.356261 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.352239 25147 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.356868 25147 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.356858 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.357342 25147 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.356727 25148 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.357900 25148 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.358000 25190 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.358168 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.358353 25170 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.358255 25148 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.357919 25105 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.359035 25170 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.359400 25145 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.359460 25148 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.360152 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.360189 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.360975 25105 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.361074 25190 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.361831 25190 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:39.343591 25169 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.359355 25170 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.363091 25170 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.363554 25190 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.366281 25169 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.361920 25147 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.368602 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.368880 25147 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240426 13:18:39.369890 25147 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
W20240426 13:18:39.369762 25070 leader_election.cc:341] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.370676 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.371266 24955 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.371812 25148 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20240426 13:18:39.373045 25148 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.378639 25147 tablet_bootstrap.cc:654] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.385052 24851 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.390064 25170 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.392035 25147 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.393239 25147 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.001s
I20240426 13:18:39.393146 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.393790 25030 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.395757 25147 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.396561 25147 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.396826 25147 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.397449 25147 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.345810 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.419342 25190 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.420301 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.420907 25105 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.421468 25147 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20240426 13:18:39.424789 25147 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.408603 24955 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:39.414525 25149 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.415807 25148 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:39.419440 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73751.1us
I20240426 13:18:39.427593 25149 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.430863 25105 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.431465 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.432359 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:39.433493 25157 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.403506 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.426863 25170 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.446695 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.447966 25169 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.448549 25169 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:39.449522 25169 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.444358 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.458228 24954 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.432354 24956 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:39.473086 25197 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240426 13:18:39.477197 24854 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.479605 25190 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.486850 25149 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.489054 24954 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.489773 25166 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.490182 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.490231 25166 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.485141 25157 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.492271 25145 tablet_bootstrap.cc:492] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.494247 25145 ts_tablet_manager.cc:1367] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.150s user 0.045s sys 0.030s
I20240426 13:18:39.495848 25157 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.492786 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:39.498543 25166 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:39.501444 25070 leader_election.cc:341] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.504119 25145 raft_consensus.cc:348] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.504810 25145 raft_consensus.cc:374] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.505076 25145 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.505738 25145 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.506733 25149 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.507198 25149 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.507774 25145 ts_tablet_manager.cc:1398] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240426 13:18:39.507627 25149 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.509805 25147 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.512012 25166 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.512588 25166 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.520676 25216 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.522079 25145 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.522433 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.524567 25197 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 13:18:39.521936 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.521644 25216 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.535089 25166 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.535807 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.536468 25216 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.537443 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.537478 25069 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.537950 25029 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.539119 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.539911 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.539121 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:39.541194 24921 leader_election.cc:341] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.544199 25216 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.545271 25216 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.545482 25030 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:39.545840 25149 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.547329 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.548610 25166 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.548713 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:39.549432 24954 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.568647 25148 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.569247 25148 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.196s user 0.032s sys 0.009s
I20240426 13:18:39.573658 25149 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:39.577349 24954 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:39.578584 25148 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.579224 25148 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.579322 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:39.579478 25148 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.580147 25149 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.580747 25149 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:39.580344 25148 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.581001 25190 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.579918 25166 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.581598 25149 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.576254 25216 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.585345 25145 tablet_bootstrap.cc:654] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.590159 25148 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.021s user 0.006s sys 0.002s
I20240426 13:18:39.592388 25148 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.606209 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:18:39.599524 25070 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.613294 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.618039 25216 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.619879 25216 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.627136 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.622081 25070 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:39.628458 24921 leader_election.cc:341] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.628849 25169 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:39.629231 25147 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.629804 25147 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.208s user 0.014s sys 0.023s
W20240426 13:18:39.629948 25070 leader_election.cc:341] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.631924 24851 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.632951 25147 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.633827 25147 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.634133 25147 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.634873 25147 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.636667 25190 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.638067 25147 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240426 13:18:39.639043 25147 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.639621 25148 tablet_bootstrap.cc:654] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.637919 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.642266 25031 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.651718 25148 tablet_bootstrap.cc:492] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.652192 25148 ts_tablet_manager.cc:1367] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.010s
I20240426 13:18:39.654444 25147 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.654577 25148 raft_consensus.cc:348] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.655256 25148 raft_consensus.cc:374] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.655503 25148 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.656179 25148 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.657410 25031 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.658571 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.659694 25157 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.660171 25157 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.661013 25157 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.655622 25216 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.661938 25216 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.663695 25216 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.665010 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.665380 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.665639 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.665979 25104 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.666782 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.671437 25216 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.672816 25216 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.673177 25216 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.673485 25216 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.676005 25170 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.676774 25148 ts_tablet_manager.cc:1398] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240426 13:18:39.676610 25170 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.677816 25148 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.686899 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.687713 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.689083 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.690099 25197 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.690544 25197 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.690956 25197 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.694931 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:39.696671 24921 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.698493 25148 tablet_bootstrap.cc:654] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.704151 25170 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.706841 25147 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.707242 25147 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.014s
I20240426 13:18:39.709653 25147 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.710301 25147 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.710602 25147 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.710728 24851 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:39.711870 25197 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.712917 25145 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.713438 25145 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.007s
I20240426 13:18:39.714293 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.714833 25031 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.715818 25145 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.716414 25145 raft_consensus.cc:374] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.716507 25148 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.716660 25145 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.716881 25148 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.010s
I20240426 13:18:39.719162 25148 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.719936 25148 raft_consensus.cc:374] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.720257 25148 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.719839 25216 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.720890 25148 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.721179 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.721647 24854 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.722493 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.723119 25031 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.723230 25148 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:39.722812 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.723886 25157 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.724191 25148 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.724404 25157 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.726843 25216 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.727350 25157 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.726189 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.723688 25145 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.728706 25105 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.730249 25145 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240426 13:18:39.731209 25145 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.734900 25105 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.735890 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:39.736714 25216 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.737293 25216 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.738092 25216 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.741353 25148 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.744869 25197 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.745913 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:18:39.747123 24921 leader_election.cc:341] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.731951 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.760048 25190 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.760560 25190 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.764187 25157 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.764722 25157 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.782513 25145 tablet_bootstrap.cc:654] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.782346 25147 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.783720 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.784755 25147 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.077s user 0.017s sys 0.019s
W20240426 13:18:39.785583 24995 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.787456 25157 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.785634 25147 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.791546 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.791878 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.792512 25105 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.793606 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:39.794327 25157 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.794734 25157 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.795089 25157 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.796149 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.796744 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
W20240426 13:18:39.797909 24921 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.797971 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:39.798882 25193 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.799224 25193 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.799286 25197 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.799707 25193 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.806793 25193 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.809283 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:39.810549 24995 leader_election.cc:341] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.814692 25147 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.823194 25147 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.823624 25147 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.004s
I20240426 13:18:39.824625 25190 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.826086 25147 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.826941 25147 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.826702 25155 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.827224 25147 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.826720 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.827710 25105 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.832099 25193 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.827896 25147 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.833827 25157 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.834913 25147 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240426 13:18:39.837164 25147 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.837656 25145 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:39.838099 25145 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.001s
I20240426 13:18:39.838466 24851 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.835065 25105 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:39.840298 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.841456 24996 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:39.842300 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.842545 25190 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.843011 25190 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:18:39.843449 24921 leader_election.cc:341] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:39.843753 25190 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.844617 25147 tablet_bootstrap.cc:654] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.840411 25145 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.845438 25145 raft_consensus.cc:374] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.846638 25145 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.847651 25145 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.855610 25148 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:39.857729 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.859201 25148 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.135s user 0.023s sys 0.011s
I20240426 13:18:39.859341 25104 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.857059 25157 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.858750 25155 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:39.870215 25104 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.871680 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:39.872673 25157 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:39.873279 25157 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:39.874315 25157 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.874408 25031 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.893290 25145 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20240426 13:18:39.894847 25145 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:39.894457 25235 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:39.903239 25148 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.904124 25148 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.913378 25148 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.915511 25155 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.915956 25155 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.911024 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.920338 25155 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.924634 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.921937 25148 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.929010 25147 tablet_bootstrap.cc:492] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:39.929495 25147 ts_tablet_manager.cc:1367] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.092s user 0.027s sys 0.025s
I20240426 13:18:39.929822 25148 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20240426 13:18:39.930648 25148 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:39.933108 25157 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.934105 25147 raft_consensus.cc:348] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.934758 25147 raft_consensus.cc:374] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:39.935025 25147 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:39.935969 25145 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.935658 25147 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.939220 25235 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.939849 25235 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.942399 25235 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.944087 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:39.944698 25029 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.945242 25147 ts_tablet_manager.cc:1398] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240426 13:18:39.945767 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:39.946117 25147 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:39.946739 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.944962 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:39.947714 25157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:39.949117 25157 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:39.949452 25157 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.949486 24995 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:39.948559 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:39.959249 25157 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.967674 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:39.968475 25029 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.970281 25147 tablet_bootstrap.cc:654] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.972028 24854 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:39.979316 25148 tablet_bootstrap.cc:654] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:39.982671 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:39.983428 25105 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:39.984706 25235 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.987731 25190 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:39.981370 25157 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.988319 25190 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:39.990728 25190 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:39.992883 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:39.993645 24956 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:39.995702 25029 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:39.996922 25149 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:39.997033 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.001345 25105 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.004864 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.006505 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.008394 25247 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.008453 25235 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.008891 25235 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.041021 25147 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.041471 25147 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.008s
I20240426 13:18:40.001773 25157 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.041290 25155 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:40.052523 25157 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.052981 25247 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.053735 25147 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.054438 25147 raft_consensus.cc:374] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.054754 25147 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.053982 25157 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.055342 25147 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.066810 25147 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240426 13:18:40.067641 25147 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.072167 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.070062 25216 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240426 13:18:40.073866 25147 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:40.076319 24921 leader_election.cc:341] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.070597 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.064723 25235 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:40.085969 24921 leader_election.cc:341] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.088665 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.089362 25145 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.089773 25145 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.195s user 0.019s sys 0.003s
I20240426 13:18:40.092224 25145 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.092501 25166 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.092972 25145 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.093225 25145 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.093976 25145 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:40.095149 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:40.096447 25145 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240426 13:18:40.096957 25148 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.097347 25148 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.018s
I20240426 13:18:40.097362 25145 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.099581 25148 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.100226 25148 raft_consensus.cc:374] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.102558 25148 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.103498 25148 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.108045 25166 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.109156 25166 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.113283 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.116235 25157 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.118233 25145 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.123868 25148 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20240426 13:18:40.125569 25148 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.117949 25166 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.127606 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.128383 25029 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:18:40.130872 25070 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:40.131994 25070 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.133627 25166 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.140956 24854 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.153517 25145 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.153821 25145 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.004s
I20240426 13:18:40.154434 25148 tablet_bootstrap.cc:654] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.158152 25166 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.159952 25166 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.160451 25145 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.161128 25145 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.161356 25145 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.161988 25145 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.165258 25145 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20240426 13:18:40.166500 25147 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.167305 25147 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.020s
I20240426 13:18:40.169277 25147 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.169679 25147 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.169835 25147 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.171588 25145 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.173722 25148 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.174275 25148 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.002s
I20240426 13:18:40.170375 25147 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.177080 25148 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.177819 25148 raft_consensus.cc:374] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.178121 25148 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.178856 25148 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.180938 25148 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:40.182272 25148 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.182942 25145 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.189955 25148 tablet_bootstrap.cc:654] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.197451 25145 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.197883 25145 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.008s
I20240426 13:18:40.199543 25148 tablet_bootstrap.cc:492] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.177080 25147 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240426 13:18:40.199944 25148 ts_tablet_manager.cc:1367] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20240426 13:18:40.200626 25147 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.200331 25145 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.201045 25145 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.201328 25145 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.201993 25145 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.202204 25148 raft_consensus.cc:348] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.203003 25148 raft_consensus.cc:374] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.203265 25148 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.204205 25145 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240426 13:18:40.204021 25148 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.206130 25145 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.206233 25147 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.206259 25148 ts_tablet_manager.cc:1398] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:40.207602 25148 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.215765 25148 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.216056 25235 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.217695 25145 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.222275 25216 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.224406 25148 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.224823 25148 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20240426 13:18:40.227622 25148 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.228425 25148 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.228735 25148 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.230643 25148 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.233117 25148 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 13:18:40.233934 25148 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.234972 25216 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.238567 25145 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.239166 25145 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.011s
I20240426 13:18:40.239367 25148 tablet_bootstrap.cc:654] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.241741 25145 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.243047 25145 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.243398 25145 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.245234 25155 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.245878 24954 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.244096 25145 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.254128 25148 tablet_bootstrap.cc:492] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.254719 25148 ts_tablet_manager.cc:1367] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.006s
I20240426 13:18:40.256763 25216 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.257099 25145 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240426 13:18:40.257381 25148 raft_consensus.cc:348] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.258276 25148 raft_consensus.cc:374] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.258544 25148 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.259325 25145 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.259191 25148 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.262038 25148 ts_tablet_manager.cc:1398] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240426 13:18:40.263146 25155 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.265477 25148 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.269491 25235 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.273002 25166 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.273459 25166 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.273967 25148 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.275761 25166 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.277621 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.278322 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.278780 25069 leader_election.cc:341] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.266660 25147 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.279320 25155 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.279592 25147 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.079s user 0.013s sys 0.000s
I20240426 13:18:40.279908 25155 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.282241 25155 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.282946 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.283324 25070 leader_election.cc:341] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.283737 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.284454 25155 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.284793 25166 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.284991 25155 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.287688 25145 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.287869 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.288908 25155 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.289151 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.289408 24996 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.289814 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.289300 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.290812 25101 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.290947 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.291798 25247 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.292518 25247 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.292876 25247 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.293177 25247 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.291838 25166 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.296702 25148 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.297284 25148 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.001s
I20240426 13:18:40.296636 25166 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.299728 25148 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.300072 25170 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.300443 25148 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.300817 25148 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.300691 25170 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.301524 25148 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.303001 25170 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.303795 25101 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.304272 25148 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240426 13:18:40.303917 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.305157 25148 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.305778 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.306423 25166 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.308437 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.308607 25070 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.309033 25031 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.309672 25216 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.310089 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.310721 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.311580 25148 tablet_bootstrap.cc:654] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.281973 25147 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.312723 25147 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.313007 25147 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.314709 25247 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.316347 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.320228 25170 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.320650 25170 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.321048 25170 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.326287 25235 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.332510 25029 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.340567 25247 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.341812 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.342665 25102 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.344521 25235 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:40.352427 25169 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.360929 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.361809 24956 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.363226 25145 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.363855 25145 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.004s
I20240426 13:18:40.366466 25145 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.367336 25145 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.367652 25145 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.369256 25145 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.370864 25102 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.372366 24996 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.373297 25247 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.373915 25247 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.374899 25247 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.373297 25145 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240426 13:18:40.383997 25145 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.341871 25170 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.395993 25145 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.404136 25147 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.407166 25147 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.127s user 0.006s sys 0.000s
I20240426 13:18:40.418309 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.419116 25029 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.429344 25029 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.430722 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.431866 25235 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.432476 25235 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.433981 25169 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.433517 25235 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.434443 25169 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.437198 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.437822 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.439149 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:40.440229 25166 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.440672 25166 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.441094 25166 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.444459 25169 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.445188 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.445757 25101 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.446707 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.447820 25170 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.447930 25216 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240426 13:18:40.447840 25070 leader_election.cc:341] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.453359 25166 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.456909 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.457659 24954 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.460290 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:40.461829 25070 leader_election.cc:341] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.463441 25166 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.470988 25148 tablet_bootstrap.cc:492] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.471458 25148 ts_tablet_manager.cc:1367] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.166s user 0.071s sys 0.086s
I20240426 13:18:40.473783 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.474848 25148 raft_consensus.cc:348] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.475497 25148 raft_consensus.cc:374] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.475757 25148 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.476400 25247 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.476492 25148 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.476927 25247 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.478549 24954 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.479365 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.480386 24995 leader_election.cc:341] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.480782 25197 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.481034 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.482650 25147 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.482702 25247 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.483410 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.483951 25166 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.484622 25166 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:18:40.484750 24996 leader_election.cc:341] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.485239 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.485932 25247 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.486687 25166 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.478554 25148 ts_tablet_manager.cc:1398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240426 13:18:40.488983 25145 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.489429 25145 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.026s
I20240426 13:18:40.491372 25147 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.488981 25148 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.495651 24854 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.494333 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.500017 25145 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.500689 25145 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.500924 25145 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.501382 25145 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.505240 25145 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240426 13:18:40.506090 25145 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.512100 25145 tablet_bootstrap.cc:654] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.515201 25148 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.527693 25147 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.528123 25147 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.009s
I20240426 13:18:40.529464 24854 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.530334 25147 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.531018 25147 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.531286 25147 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.531916 25147 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.532783 25145 tablet_bootstrap.cc:492] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.533210 25145 ts_tablet_manager.cc:1367] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.002s
I20240426 13:18:40.533890 25148 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.534101 25147 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:40.534300 25148 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.000s
I20240426 13:18:40.534966 25147 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.535310 25145 raft_consensus.cc:348] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.535916 25145 raft_consensus.cc:374] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.536172 25145 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.536774 25145 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.537179 25148 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.537971 25148 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.538203 25148 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.538697 25145 ts_tablet_manager.cc:1398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240426 13:18:40.538794 25148 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.539589 25145 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.541211 25148 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240426 13:18:40.542178 25148 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.546891 25148 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.547899 25147 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.553467 25145 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.567313 25145 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.567744 25145 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.000s
I20240426 13:18:40.568281 25147 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.568730 25147 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.003s
I20240426 13:18:40.568760 25148 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.569339 25148 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.018s
I20240426 13:18:40.570180 25145 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.570852 25145 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.571142 25145 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.571043 25147 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.571944 25147 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.571805 25148 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.572466 25147 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.571868 25145 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.572777 25148 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.573135 25148 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.573174 25147 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.574008 25148 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.574471 25145 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240426 13:18:40.575902 25145 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.576424 25148 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240426 13:18:40.577334 25148 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.578771 24854 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:40.581605 25145 tablet_bootstrap.cc:654] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.584524 25148 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.583725 25147 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240426 13:18:40.587968 25235 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.588663 25235 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.591151 25235 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.594704 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.595407 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.596393 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.597126 25101 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.598973 25148 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.599481 25148 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20240426 13:18:40.599443 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:40.601434 25235 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.601940 25235 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.602363 25235 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.602598 25148 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.603466 25148 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.603794 25148 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.604460 25148 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.606576 25148 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240426 13:18:40.607470 25148 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.612520 25148 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.612823 25147 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.618116 25235 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.623744 25148 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.624320 25148 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20240426 13:18:40.626701 25148 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.627389 25148 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.627736 25148 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.628538 25148 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.630847 25148 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 13:18:40.631542 25147 tablet_bootstrap.cc:654] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.631778 25148 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.635936 25148 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.637858 25235 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.638482 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.639209 25031 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.641424 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.642028 25101 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.649012 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.654670 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.649497 25101 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.655723 25235 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.656247 25235 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.657054 25235 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.651461 25145 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.659993 25145 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.084s user 0.033s sys 0.026s
I20240426 13:18:40.660487 25148 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.660944 25148 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.013s
I20240426 13:18:40.652336 25147 tablet_bootstrap.cc:492] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.662866 25147 ts_tablet_manager.cc:1367] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.007s
I20240426 13:18:40.663483 25148 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.663467 25145 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.664425 25148 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.664600 25145 raft_consensus.cc:374] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.664758 25148 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.665048 25145 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.665513 25147 raft_consensus.cc:348] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.665513 25148 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.666410 25147 raft_consensus.cc:374] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.665863 25145 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.666756 25147 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.667559 25147 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.668136 25148 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240426 13:18:40.670262 25147 ts_tablet_manager.cc:1398] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:18:40.671156 25147 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.673017 25148 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.674952 25145 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20240426 13:18:40.675976 25145 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.676662 25147 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.679224 25148 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.681105 25145 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.696202 25148 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.696691 25148 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20240426 13:18:40.698279 25145 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.698673 25145 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.005s
I20240426 13:18:40.699081 25148 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.699746 25148 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.699985 25148 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.700385 25145 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.700906 25145 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.700593 25148 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.701121 25145 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.702044 25145 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.703759 25145 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240426 13:18:40.705016 25145 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.706192 25148 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240426 13:18:40.707204 25148 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.707520 25147 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.707988 25147 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.002s
I20240426 13:18:40.708178 25031 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.708798 25247 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.709250 25247 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.711102 25247 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.711061 25147 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.711992 25147 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.712319 25147 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.712833 25216 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.712994 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.713068 25147 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.713302 25216 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.713685 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.712419 25148 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.715209 25216 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.717391 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.717974 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.717880 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.718757 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.719357 25101 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.720111 25155 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.720230 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.720659 25155 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.720927 25104 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.721177 25155 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.723526 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:40.725760 25235 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.726233 24851 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.726261 25235 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.728690 25235 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.731634 25145 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.743014 25247 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.742836 25235 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.743891 25247 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.745754 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.746351 25029 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.747253 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.748961 24995 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.750042 25145 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.750458 25145 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.000s
I20240426 13:18:40.752832 25145 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.753409 25145 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.753233 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.753708 25145 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
W20240426 13:18:40.754976 24996 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.755523 24996 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.756568 25190 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.758208 25247 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.759546 25145 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.762542 25155 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.768524 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.769228 24956 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.770103 25145 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240426 13:18:40.763767 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.773460 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.774204 25216 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.774806 25216 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:40.775161 25145 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.775861 25216 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.776497 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.778789 25104 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.779661 25235 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.780457 25145 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.780407 25235 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.780920 25235 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.784471 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.785590 24921 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.786252 25166 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.786674 25166 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.788525 25166 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.790225 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.790747 25247 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.791532 25247 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.792484 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.790860 24955 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.794569 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:40.795434 25169 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.796039 25169 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.796254 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.797044 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:40.797930 25169 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.798154 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.798453 25169 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.798791 25169 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.798959 25190 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.799114 25169 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.799340 25190 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:18:40.799374 25070 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.799768 25190 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.801219 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.802757 25069 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.803586 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.804711 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:18:40.805542 24995 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.806213 25169 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.795470 25247 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.807937 25190 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.808789 25169 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.809175 25030 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:40.810133 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.811151 25101 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:40.813611 24921 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.814122 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.814723 25216 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:40.815850 24995 leader_election.cc:341] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.820947 24956 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.823817 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.826807 25197 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.841540 25155 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.841269 25197 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.842921 25197 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.842355 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.827441 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.845081 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.845628 25155 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.845964 25155 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.846637 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.846760 25155 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.830765 25169 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.848351 25169 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.848853 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.849510 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.850068 25169 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.850795 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.851652 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.852803 25101 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.838618 25148 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.838416 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.856319 25148 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.149s user 0.055s sys 0.055s
W20240426 13:18:40.857108 25070 leader_election.cc:341] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.858335 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.858919 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:40.829435 25235 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.840507 25190 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.845877 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.859774 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:40.860002 25102 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.860265 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.765748 25147 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20240426 13:18:40.852206 25105 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:40.831787 25216 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.861809 25166 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.862783 25166 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.863168 25166 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.863862 25190 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.864322 25101 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.864289 25190 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.864894 25169 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.840401 25104 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:40.866197 25190 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.866843 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.858659 25148 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.868152 25148 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.868407 25148 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.868782 25145 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.869158 25145 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.013s
I20240426 13:18:40.868927 25148 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.870903 25148 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
W20240426 13:18:40.869259 24995 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.871771 24955 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.851914 25170 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:40.872200 25170 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:40.872517 25170 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.872416 25166 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.872727 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:40.873947 25190 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.874536 25190 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:40.875064 25147 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:40.874946 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:40.875452 25190 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.877760 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:40.879205 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.879741 25029 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:40.880189 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.881001 25169 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:18:40.880203 24996 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.881500 25169 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:40.881644 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:40.882365 25247 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.882318 25169 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.884588 25148 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.885758 25247 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.889075 24853 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.878312 25170 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.890535 24956 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:18:40.891220 25170 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:40.878358 25166 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.895738 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:40.896451 25104 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:40.896198 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:40.878938 25102 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:40.871464 25145 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.898070 25145 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.898414 25145 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.899735 25145 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.902070 25145 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240426 13:18:40.903087 25145 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:40.889575 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:40.904873 25148 tablet_bootstrap.cc:654] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.905929 25070 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:40.908339 25220 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:40.908814 25220 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:40.909550 25220 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.909706 25147 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.915239 25145 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.905994 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.928212 25170 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.927505 24851 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:40.929193 25170 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.936061 25147 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:40.936451 25147 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.000s
I20240426 13:18:40.938715 25147 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.939385 25147 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.939663 25147 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.940951 25147 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.955168 25247 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.958863 25148 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:40.959290 25148 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.004s
I20240426 13:18:40.961849 25148 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.962759 25148 raft_consensus.cc:374] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.963011 25148 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.963765 25145 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:40.964181 25145 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.008s
I20240426 13:18:40.963882 25148 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.967762 25145 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.968475 25145 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:40.968811 25145 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:40.969276 25216 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:40.969336 25145 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.970815 25220 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.971239 25220 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.973232 25148 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20240426 13:18:40.975678 25145 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240426 13:18:40.977852 25148 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:40.977963 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:40.982308 25220 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:40.983132 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:40.984486 25069 leader_election.cc:341] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.985616 25147 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20240426 13:18:40.989013 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.990278 25145 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:40.990602 25070 leader_election.cc:341] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.991036 25070 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:40.992112 25169 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:40.993522 25220 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:40.994648 25148 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:40.993909 25220 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:40.997761 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:40.997983 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:40.998490 25030 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:40.998916 25069 leader_election.cc:341] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:40.999449 25070 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.000044 25169 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.000386 25169 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.000695 25169 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.004128 25145 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.006382 25220 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.006752 25169 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.008138 25147 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.015027 25145 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.015422 25145 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.011s
I20240426 13:18:41.015779 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:41.017109 25069 leader_election.cc:341] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.028052 24853 catalog_manager.cc:5484] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.149998 25145 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.150823 25145 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.151129 25145 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.152467 25145 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.150094 25220 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.028630 25169 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.028692 25155 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.058871 25247 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.157166 25145 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.141s user 0.003s sys 0.005s
W20240426 13:18:41.134335 25070 consensus_peers.cc:485] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:41.157750 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.156664 25220 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.161954 25030 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.160249 25170 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:41.163118 25148 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.163347 25220 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.163549 25148 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.188s user 0.008s sys 0.021s
I20240426 13:18:41.164553 25147 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.164678 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.165861 25148 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.166312 25148 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.166509 25148 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.166937 25148 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.164925 25220 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.164746 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.169762 25028 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:41.170423 25069 leader_election.cc:341] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.172910 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.158344 25155 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.174006 25149 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.174374 25149 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.174681 25149 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.176255 25170 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.180300 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:41.181798 24995 leader_election.cc:341] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.184167 25155 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.184710 25220 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.156323 25247 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.185266 25220 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.189108 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:41.189196 25166 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.190804 25145 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.189824 25166 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.191942 25220 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.192759 25166 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.193660 25190 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.193248 25030 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.200814 25148 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.037s user 0.002s sys 0.004s
I20240426 13:18:41.201457 25148 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.200080 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.188364 25149 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.200156 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.200376 25190 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.206504 24955 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.206490 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.214058 25247 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.216058 25149 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.216748 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.217051 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.218024 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:18:41.218475 24996 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:41.219409 25069 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.219575 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.220068 25149 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.220495 25070 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:18:41.220712 25069 leader_election.cc:341] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.221297 25169 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.221776 25169 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.221993 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.222328 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.223088 24955 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.220417 25149 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.222579 25169 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.223393 24995 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.212329 25190 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.223816 24995 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.225813 25149 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.225719 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.227402 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.227919 25029 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.224505 25190 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:18:41.228291 25070 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.228588 25190 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.228675 25070 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.230979 25220 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.218827 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.233520 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.235281 25029 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:18:41.244524 25070 leader_election.cc:341] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.236011 25149 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:41.245149 24996 leader_election.cc:341] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.252104 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.251101 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.255431 25247 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.268959 25190 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.259950 25169 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.276728 25169 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.276414 24851 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.277536 25169 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.278508 25148 tablet_bootstrap.cc:654] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.280805 25149 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.281342 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.282371 24955 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.283762 25145 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.287137 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.288573 24955 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:18:41.288695 25070 leader_election.cc:341] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.289389 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.290457 25149 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.290879 25149 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.293215 25149 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.297178 25147 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.297608 25147 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.290s user 0.010s sys 0.010s
I20240426 13:18:41.302521 25190 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.312075 25147 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.312657 25147 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.312880 25147 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.313819 25147 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.315676 25147 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240426 13:18:41.316519 25147 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.325455 25247 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.323011 24854 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.333613 24854 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.335285 25147 tablet_bootstrap.cc:654] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.336196 25170 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.337585 25148 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.337365 25170 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.337911 25145 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.337977 25148 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.137s user 0.021s sys 0.017s
I20240426 13:18:41.338416 25145 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.148s user 0.006s sys 0.009s
I20240426 13:18:41.362159 25145 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.362854 25145 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.363107 25145 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.363652 25145 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.365806 25145 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.026s user 0.003s sys 0.004s
I20240426 13:18:41.366384 25147 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.366732 25147 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.003s
I20240426 13:18:41.367841 25145 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.369894 25147 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.370523 25147 raft_consensus.cc:374] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.370776 25147 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.371393 25147 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.373333 25147 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:41.374089 25147 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.373936 25148 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.374745 25148 raft_consensus.cc:374] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.375011 25148 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.376355 25148 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.379895 25147 tablet_bootstrap.cc:654] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.382853 25169 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.384121 25170 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.384548 25170 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.386385 25170 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.387691 25148 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20240426 13:18:41.388571 25148 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.391456 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.391571 25197 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.392086 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.392237 25197 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.394922 25197 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.395488 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.396275 25102 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.397286 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:41.398607 25235 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.398958 25235 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.399284 25235 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.403189 25145 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.405532 25235 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.407358 25235 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.408182 25148 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.408778 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.411063 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.411882 25102 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:41.422925 24954 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.430409 25169 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.435308 25145 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.435941 25145 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.000s
I20240426 13:18:41.439132 25145 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.440057 25145 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.440449 25145 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.440939 25169 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.441399 25145 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.445016 25145 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240426 13:18:41.446149 25145 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.452896 25145 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.469714 24921 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.468341 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:41.471462 25104 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.471305 25247 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.468051 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.477479 25220 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.479184 25029 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.489846 25147 tablet_bootstrap.cc:492] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.490876 25147 ts_tablet_manager.cc:1367] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.117s user 0.031s sys 0.053s
I20240426 13:18:41.494334 25148 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.494783 25148 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.106s user 0.016s sys 0.028s
I20240426 13:18:41.497141 25148 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.498032 25148 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.498454 25148 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.501317 25148 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.516598 25148 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20240426 13:18:41.518278 25147 raft_consensus.cc:348] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.519646 25147 raft_consensus.cc:374] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.520025 25147 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.521679 25147 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.532348 25147 ts_tablet_manager.cc:1398] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.041s user 0.006s sys 0.004s
I20240426 13:18:41.534694 25147 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.513137 25220 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.559059 25197 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.559645 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
W20240426 13:18:41.560634 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79614.5us
I20240426 13:18:41.546164 25220 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.562494 25155 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240426 13:18:41.570395 25220 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.573150 25104 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:41.575251 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:41.577991 25145 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.578459 25145 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.133s user 0.016s sys 0.043s
I20240426 13:18:41.582105 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.583252 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.583950 25028 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.579252 25340 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
W20240426 13:18:41.585706 25069 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.585800 25340 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:41.591818 25070 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.593039 25220 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.593473 25220 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.593842 25220 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.594214 25148 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.603330 25145 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.603797 25145 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.603979 25145 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.604558 25145 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.610224 25345 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.616875 25345 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.618140 25145 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20240426 13:18:41.619038 25145 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.622303 25147 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.600277 25340 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.620180 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.629910 25028 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.626375 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.619426 25345 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.633791 25148 tablet_bootstrap.cc:654] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.634008 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.637895 25193 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.638334 25193 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.635381 25197 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.642714 25193 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.643332 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:41.644822 24995 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.647030 25220 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.650235 25145 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.652089 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:41.653506 24996 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.653900 24996 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.654701 25193 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.654850 25197 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.669916 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.670852 25147 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.671489 25147 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.008s
W20240426 13:18:41.671371 25069 leader_election.cc:341] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.670024 25220 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.673110 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.673718 25028 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.673550 25147 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.674250 25147 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.674500 25147 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.674891 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:41.675395 25147 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.678347 25147 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240426 13:18:41.679320 25147 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.679373 25220 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.680063 25028 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.681800 25145 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.681867 25148 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.682369 25145 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.006s
I20240426 13:18:41.682454 25148 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.088s user 0.016s sys 0.010s
I20240426 13:18:41.684522 25147 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.684316 25070 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.684768 25145 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.685447 25145 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.685427 25149 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.685235 25148 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.686064 25148 raft_consensus.cc:374] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.686364 25148 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.686090 25149 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.687135 25148 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.687508 25149 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.692888 25148 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 13:18:41.693986 25148 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.700773 25148 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.706382 25147 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.706805 25147 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20240426 13:18:41.707319 25148 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.707930 25148 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240426 13:18:41.709218 25147 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.710088 25147 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.710441 25147 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.710225 25148 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.711155 25148 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.711491 25148 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.711354 25147 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.712381 25148 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.714244 25148 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 13:18:41.715088 25148 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.717595 25147 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240426 13:18:41.718703 25147 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.720316 25148 tablet_bootstrap.cc:654] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.721779 25145 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.723853 25145 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.727021 25147 tablet_bootstrap.cc:654] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:41.729597 25069 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:41.731248 25145 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20240426 13:18:41.735184 25145 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.740248 25247 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.740855 25247 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.744524 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.746098 25145 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.747651 25148 tablet_bootstrap.cc:492] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.748153 25148 ts_tablet_manager.cc:1367] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.011s
I20240426 13:18:41.748531 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.749290 24955 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:41.750243 24995 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.751045 25155 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.750538 25148 raft_consensus.cc:348] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.751576 25148 raft_consensus.cc:374] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.750651 24853 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.751924 25148 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.752784 25148 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.743510 25247 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.757449 25155 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.757807 25149 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.761888 25148 ts_tablet_manager.cc:1398] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240426 13:18:41.762806 25148 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.768337 25148 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.774160 25166 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:41.788384 25340 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.789005 25340 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.796998 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.791549 25340 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:41.809458 24921 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.823508 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.825392 25028 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:41.832304 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.833583 25340 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.833956 25340 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.834256 25340 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.834868 25148 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.835309 25148 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.073s user 0.023s sys 0.047s
I20240426 13:18:41.837270 25147 tablet_bootstrap.cc:492] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.837606 25145 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:41.838047 25145 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.103s user 0.032s sys 0.011s
I20240426 13:18:41.837641 25147 ts_tablet_manager.cc:1367] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.119s user 0.021s sys 0.024s
I20240426 13:18:41.841795 25147 raft_consensus.cc:348] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.842518 25147 raft_consensus.cc:374] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.842763 25147 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.847433 25147 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.849298 25147 ts_tablet_manager.cc:1398] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240426 13:18:41.840337 25145 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.849807 25145 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.850100 25145 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.850783 25145 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.854426 25147 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:41.853302 25340 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.855470 25149 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.856010 25149 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.857862 25340 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.858342 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.858711 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:18:41.859628 25069 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.859831 25145 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20240426 13:18:41.859982 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.857564 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:41.860381 24921 leader_election.cc:341] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.860728 25030 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.858732 25149 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:18:41.861300 25070 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.862043 25070 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.861235 25148 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.863561 25148 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.863792 25148 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.864405 25169 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.864889 25147 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.866029 25148 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.867728 25169 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.868453 25148 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240426 13:18:41.869320 25148 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.871038 25030 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:41.860788 25145 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:41.872087 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.872943 25340 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.873368 25340 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:41.874126 25340 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.879567 25148 tablet_bootstrap.cc:654] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.880535 25155 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.881052 25155 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.887392 25166 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.887856 25166 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.889627 25166 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.891518 25149 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:41.892776 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.912459 25145 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.893164 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.893164 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.913138 25028 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.913393 25345 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.912743 24956 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:41.920789 25345 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.924237 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:41.925336 25149 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.929915 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.931906 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:41.932155 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.932643 25105 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
W20240426 13:18:41.933833 24921 leader_election.cc:341] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.925283 25345 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.935649 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:41.937670 25148 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.926049 25155 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:41.928503 25149 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.938803 25148 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.005s
I20240426 13:18:41.939082 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:41.939657 25149 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.939868 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:41.942209 25155 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:41.947295 25149 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.947834 25148 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.948765 25148 raft_consensus.cc:374] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.949088 25148 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.949896 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:41.950630 24955 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.950261 25148 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.952414 25346 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.952621 25148 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240426 13:18:41.952806 25346 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.953171 25346 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.953787 25148 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.964744 25147 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:41.965226 25147 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.003s
I20240426 13:18:41.966993 24955 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.967974 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:41.968451 25148 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:41.968762 25220 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:41.969244 25220 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:41.970062 25220 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.975574 25169 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:41.976016 25169 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.977941 25169 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.978246 25149 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:41.979948 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.980535 25030 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.980707 25147 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.981421 25147 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.981573 25148 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:41.981685 25147 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.982097 25148 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.007s
I20240426 13:18:41.982476 25147 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.984568 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:41.984457 25148 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.985200 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:41.986433 25148 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:41.986799 25031 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
W20240426 13:18:41.986830 25069 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:41.986830 25148 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:41.987933 25070 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:41.988271 25148 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.989219 25220 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:41.989833 25220 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:41.990353 25220 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.990418 25148 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 13:18:41.990746 25030 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:41.991555 25148 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:41.993424 25346 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:41.994720 24854 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:41.995942 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:41.996500 25030 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:41.998220 25148 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.002625 25030 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.003635 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:42.004321 25340 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.004925 25340 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:42.005555 25346 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.005697 25340 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.006631 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.007457 25102 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.009397 25147 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20240426 13:18:42.010257 25147 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.011759 25145 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.012217 25145 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.152s user 0.010s sys 0.002s
I20240426 13:18:42.014958 25145 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.015643 25145 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.016249 25145 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.016870 25145 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.019120 25145 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:18:42.021528 25145 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.020216 25220 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.027045 25220 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.028187 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.029044 25031 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.030889 25145 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.036489 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.038071 25069 leader_election.cc:341] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.041951 25031 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.042824 25145 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.043329 25145 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.012s
I20240426 13:18:42.043460 25070 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.044549 25220 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.045008 25220 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.045773 25220 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.047461 25169 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.047701 25102 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.047904 25169 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.049388 25147 tablet_bootstrap.cc:654] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.049681 25169 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.051162 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.052510 25069 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.053475 25148 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.053937 25148 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.026s
I20240426 13:18:42.053125 25145 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.055670 25340 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.055877 25145 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.056174 25340 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.056533 25148 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.056188 25145 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.057354 25148 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.057710 25148 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.058187 25340 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.058332 25148 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.058894 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.059578 25145 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.060786 25148 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240426 13:18:42.062011 25148 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
W20240426 13:18:42.061908 24921 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.062913 25145 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240426 13:18:42.062095 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.065996 25145 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.066562 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.067206 25102 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:42.053292 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.068027 25148 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.068087 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.069275 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.072872 25169 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.073249 25169 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.073614 25169 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.076699 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.077541 25343 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.078012 25343 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.078388 25343 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.082471 25169 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.087719 25147 tablet_bootstrap.cc:492] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.088183 25147 ts_tablet_manager.cc:1367] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.001s
I20240426 13:18:42.092334 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.093711 25069 leader_election.cc:341] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.094558 25169 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.097895 25170 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.100834 25145 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.111560 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.113658 25145 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.113716 25147 raft_consensus.cc:348] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.125309 25147 raft_consensus.cc:374] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.125656 25147 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.126230 25147 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.127727 25147 ts_tablet_manager.cc:1398] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20240426 13:18:42.117465 24854 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.120293 25148 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.129598 25148 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.068s user 0.034s sys 0.019s
I20240426 13:18:42.132056 25148 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.132773 25148 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.132269 25220 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.133152 25148 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.133811 25148 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.125241 25031 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.135326 25148 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240426 13:18:42.145462 25145 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.004s
I20240426 13:18:42.141443 25147 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.146385 25148 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.124555 25343 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.151275 25147 tablet_bootstrap.cc:654] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.151396 25343 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.152107 25148 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.153475 25155 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.153942 25155 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.155964 25145 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.156526 25155 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.156672 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.156652 25145 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240426 13:18:42.158128 24995 leader_election.cc:341] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.158497 25145 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.159132 25145 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.160005 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:42.163219 24921 leader_election.cc:341] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.165050 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.164501 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.165503 25145 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20240426 13:18:42.166436 25145 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.168396 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.165845 25102 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.175496 25169 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.176388 25169 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.179622 25169 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.183950 25145 tablet_bootstrap.cc:654] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.186081 25343 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.198808 25147 tablet_bootstrap.cc:492] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.199409 25147 ts_tablet_manager.cc:1367] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.022s
I20240426 13:18:42.201025 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.201858 25104 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.202441 25147 raft_consensus.cc:348] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.203296 25147 raft_consensus.cc:374] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.203660 25147 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.207271 25147 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.209010 25147 ts_tablet_manager.cc:1398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20240426 13:18:42.209251 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.209829 25147 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.210739 25247 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.211208 25247 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.211616 25247 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.220145 25155 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:42.220327 25069 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.220636 25155 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.222529 25155 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.226691 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.227596 24955 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.230598 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:42.231070 25104 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:42.232156 25155 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.232525 25155 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.232860 25155 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.235247 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.235819 25031 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.238054 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.238787 25343 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.239229 25343 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:42.240025 25343 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.241991 25155 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.235889 25102 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.247160 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.247696 24954 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.248675 25247 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.250690 25148 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.251137 25148 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.023s
I20240426 13:18:42.255249 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.256448 24995 leader_election.cc:341] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.256765 25147 tablet_bootstrap.cc:654] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.257217 25148 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.257908 25148 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.258177 25148 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.258239 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.258409 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.258806 25102 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.258844 25148 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.259748 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:42.260898 25193 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.261245 25148 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240426 13:18:42.262107 25148 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.265290 25220 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.265780 25220 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.266678 25155 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.268752 25148 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.269140 25220 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.270236 25193 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.270690 25247 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.271301 25193 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.275702 25145 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.276152 25145 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.000s
I20240426 13:18:42.276021 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.276695 25030 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.278070 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:42.278482 25145 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.279129 25145 raft_consensus.cc:374] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.279451 25145 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.280460 25145 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.282922 25145 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:42.282756 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.283465 25104 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.283799 25145 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.289796 25102 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.281190 25220 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.291026 25220 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.291409 25220 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.293951 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.295585 25388 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.295960 25388 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.296582 25388 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.297996 25220 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.290449 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.286182 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.299894 25220 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.316653 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.319972 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.320628 25029 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.313388 24954 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.379456 25388 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.379976 25388 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.381886 25388 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.378427 24854 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.384766 25147 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
W20240426 13:18:42.389254 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:42.390450 25147 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.032s
I20240426 13:18:42.388620 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.396808 24995 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.399363 25104 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.405560 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.406803 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.410882 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.415786 25145 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.418313 24996 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.418726 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.419426 25388 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.423183 25029 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.427503 25170 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.429337 25340 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.429474 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.430265 25220 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.430742 25220 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.433621 25147 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.434194 25147 raft_consensus.cc:374] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.434392 25147 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.431555 25220 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.435051 25147 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.433614 25102 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.440763 25147 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20240426 13:18:42.441574 25147 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.448122 24954 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.451390 25148 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.451999 25148 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.190s user 0.008s sys 0.026s
I20240426 13:18:42.456977 25148 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.457617 25148 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.457887 25148 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.458498 25148 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.459916 25148 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240426 13:18:42.460772 25148 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.463085 25169 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.478292 25170 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240426 13:18:42.481338 25147 tablet_bootstrap.cc:654] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.477461 25220 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240426 13:18:42.502204 25169 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.503084 25148 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.525061 25170 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.526011 25169 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.545311 25145 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.545707 25145 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.262s user 0.005s sys 0.014s
I20240426 13:18:42.547742 25145 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.559172 25145 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.559417 25145 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.568117 25147 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.551628 24851 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.570798 25147 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.005s
I20240426 13:18:42.573069 25147 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.573693 25147 raft_consensus.cc:374] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.573977 25147 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.573747 25145 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.576262 25148 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.576714 25148 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.003s
I20240426 13:18:42.577303 25145 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
I20240426 13:18:42.578153 25145 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.578876 25148 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.579551 25148 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.579789 25148 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.585341 25147 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.587152 25147 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20240426 13:18:42.587667 25148 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.589743 25148 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240426 13:18:42.590482 25148 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.592473 25147 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.595963 25247 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.596410 25247 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.598287 25247 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.599714 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.601848 25147 tablet_bootstrap.cc:654] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.602702 24995 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.600826 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.604691 25148 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:42.605141 24996 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.605551 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.606241 25247 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.611574 25145 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.612375 25147 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.612780 25147 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240426 13:18:42.614809 25147 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.615427 25147 raft_consensus.cc:374] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.615693 25147 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.616461 25147 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.617313 25340 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.617655 25145 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.618113 25145 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.001s
I20240426 13:18:42.617719 25340 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.618724 25147 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:42.619787 25147 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.620362 25145 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.621140 25145 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.621428 25145 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.621992 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.622085 25145 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.625087 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.625473 25148 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.625854 25148 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.004s
I20240426 13:18:42.626736 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:42.627389 25170 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.627821 25340 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.628650 25148 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.629297 25148 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.629546 25148 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.630064 25148 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.631742 25145 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240426 13:18:42.632575 25145 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.632690 25148 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240426 13:18:42.634799 25147 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.637684 25148 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.638214 25145 tablet_bootstrap.cc:654] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.642699 25148 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.648464 25345 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.656404 25148 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.656776 25148 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.008s
I20240426 13:18:42.678148 25147 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.678669 25147 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.010s
I20240426 13:18:42.679872 25247 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.681314 25148 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.681947 25148 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.682173 25148 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.682701 25148 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.684829 25145 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.684939 25148 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20240426 13:18:42.685333 25145 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.015s
I20240426 13:18:42.685746 25148 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.687582 25145 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.688306 25145 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.689172 25145 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.692123 25145 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.689705 25147 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.693814 25147 raft_consensus.cc:374] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.694247 25147 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.690981 25148 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.697022 25147 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.707363 25148 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.696000 25145 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240426 13:18:42.707798 25148 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20240426 13:18:42.708099 25147 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
W20240426 13:18:42.708524 24921 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Couldn't send request to peer 46106ff992984d33bb4a4d8f30162639. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.717914 25155 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.713203 24954 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.726315 25147 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.728564 24955 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.714313 25145 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.732466 25147 tablet_bootstrap.cc:654] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.732704 25155 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.733124 25155 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.734683 25145 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.735765 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.710012 25148 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.736398 25148 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.736658 25148 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.737105 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:18:42.737264 24995 leader_election.cc:341] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.738154 25103 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:42.737259 25148 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.740419 25148 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20240426 13:18:42.740173 24996 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.741287 25148 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.741178 25169 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.741336 25193 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.741942 25193 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.742373 25193 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.736979 25155 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:42.750046 24995 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:42.757997 25148 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.774295 25169 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.796499 25166 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.797024 25166 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.799322 25166 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.804931 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:42.805737 25027 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.811236 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:42.812707 25149 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.813184 25149 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.813633 25149 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.822923 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.823794 24954 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:42.826689 25193 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.831082 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:42.832201 24995 leader_election.cc:341] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.831768 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.832847 25103 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.844692 25155 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:42.849211 25149 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.853253 25145 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.859781 25145 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.146s user 0.022s sys 0.025s
I20240426 13:18:42.862677 25145 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.863314 25145 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.863569 25145 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.858209 25247 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.864395 25247 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.866407 25247 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.867038 25145 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.867658 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.867988 25147 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.868453 25147 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.142s user 0.044s sys 0.074s
W20240426 13:18:42.869027 24995 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.869694 25145 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240426 13:18:42.870597 25145 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.870729 25147 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.867337 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.871407 25147 raft_consensus.cc:374] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.871726 25147 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
W20240426 13:18:42.872553 24996 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.873278 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:42.874577 25247 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:42.875366 25193 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.874835 25103 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.880412 24996 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.881353 25388 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.881561 25193 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.882057 25193 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.881816 25388 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.882912 25193 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.884114 25388 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.884737 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:42.884989 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:42.884881 25147 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.885890 25102 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
W20240426 13:18:42.886152 24995 leader_election.cc:341] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.883878 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.887027 25147 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.018s user 0.002s sys 0.002s
I20240426 13:18:42.887074 24955 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.887936 25147 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.888549 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.889407 25422 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:42.889755 25422 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:42.890048 25422 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.893249 25147 tablet_bootstrap.cc:654] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.894179 24955 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.895392 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:42.896301 25169 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.896703 25169 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:42.897462 25169 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.902026 25148 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.903872 25147 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:42.904832 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:42.880026 25145 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.907757 25148 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.167s user 0.017s sys 0.047s
I20240426 13:18:42.906661 25149 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:42.907289 25147 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20240426 13:18:42.907306 25030 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.910032 25148 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.910648 25148 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.910930 25148 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.911548 25148 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.913761 25148 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 13:18:42.914225 25030 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:42.914525 25148 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.917009 25147 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.917619 25147 raft_consensus.cc:374] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.917881 25147 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.918433 25147 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.919829 25148 tablet_bootstrap.cc:654] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.922006 24853 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:42.923218 25147 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240426 13:18:42.923871 25147 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:42.925055 25422 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.932310 25422 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.934187 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:42.934830 25103 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.935468 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:42.942268 25145 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:42.942744 25145 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.001s
W20240426 13:18:42.945497 24995 leader_election.cc:341] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.945681 24955 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:42.948139 25148 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:42.948766 25148 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20240426 13:18:42.955313 25145 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.955924 25145 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.956212 25145 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.956247 25148 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.956813 25145 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.957423 25148 raft_consensus.cc:374] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:42.958297 25145 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240426 13:18:42.960556 25148 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:42.961217 25148 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.962433 25103 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:42.962740 25145 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:42.963585 25148 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240426 13:18:42.965626 25148 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:42.966468 25193 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:42.966888 25193 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.968623 25193 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:42.972433 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:42.973604 24995 leader_election.cc:341] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:42.973292 24852 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:42.976225 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:42.977691 25147 tablet_bootstrap.cc:654] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.979400 25169 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.982741 25388 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:42.983893 25388 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:42.984710 25388 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:42.986639 25148 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:42.987769 25169 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:42.993089 25145 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.009322 25147 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.009896 25147 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.004s
I20240426 13:18:43.016266 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.018100 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.020421 25145 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.020999 25145 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.003s
I20240426 13:18:43.023833 25147 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.024509 25145 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.025193 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.025653 25145 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.025918 25145 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.026477 25147 raft_consensus.cc:374] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.026810 25147 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.027235 25388 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.030396 25388 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.030865 25388 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.027407 25147 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.028231 25145 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.038008 25149 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.038460 25149 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.040341 25149 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.042022 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.042692 25030 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.036590 25147 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20240426 13:18:43.042881 25145 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.022s user 0.002s sys 0.004s
I20240426 13:18:43.042876 25388 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.044132 25145 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.044098 25070 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:43.044670 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.045048 25149 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.045384 25149 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.045711 25149 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.045835 25069 leader_election.cc:341] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.046787 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.048194 24995 leader_election.cc:341] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.049372 25388 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.049832 25145 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.050958 25147 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.051751 25149 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.054553 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.056191 25069 leader_election.cc:341] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.057673 25149 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.058297 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.058907 25029 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.066790 25145 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.067315 25145 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.002s
I20240426 13:18:43.070844 25343 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.071413 25343 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.073390 25343 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.070014 25145 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.074167 25145 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.074455 25145 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.075165 25145 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.075213 25029 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.077419 25145 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240426 13:18:43.077491 25070 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.078038 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.078270 25149 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.078373 25145 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.078830 25149 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.079916 25166 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.079666 25149 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.080379 25166 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.082283 25166 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.082599 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.082917 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.083173 25103 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.084200 25069 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.084156 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.085080 25030 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.086233 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.087455 25166 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.087852 25166 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.088277 25166 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.089370 25145 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.092857 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.093911 25104 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:43.095613 25147 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.099838 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.100406 25145 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.101253 25145 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20240426 13:18:43.101192 25346 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.104140 25343 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.104632 25343 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.104698 25145 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.105666 25145 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.106005 25145 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.106688 25343 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.106822 25145 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.107327 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.109674 25145 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240426 13:18:43.110863 25145 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.100755 25166 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.115630 24854 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.117048 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.118796 25069 leader_election.cc:341] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.120074 25166 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.121078 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.121654 25030 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.128545 25030 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.128700 25145 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.129575 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.130669 25166 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.131143 25166 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.131911 25166 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.133939 25346 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.139037 25103 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.141551 25145 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.142355 25145 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.007s
I20240426 13:18:43.148507 25145 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.153292 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.163915 25145 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.164498 25145 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.164817 25388 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.165309 25388 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.165521 25145 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.166146 25388 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.168188 25148 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.168642 25148 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.203s user 0.040s sys 0.031s
I20240426 13:18:43.168301 25145 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20240426 13:18:43.175088 25148 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.177415 25148 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.177651 25247 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.178167 25145 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.178094 25247 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.177698 25148 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.179976 25247 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.181398 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.184530 25145 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.181303 25148 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.185588 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.187520 25148 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s user 0.004s sys 0.001s
I20240426 13:18:43.189479 25148 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.190570 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.194870 25148 tablet_bootstrap.cc:654] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.196739 25145 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.197263 25145 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.002s
I20240426 13:18:43.200109 25145 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.201192 25145 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.201586 25145 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.202463 25145 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.202986 24852 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.204787 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.211189 25170 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.211925 25170 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.212508 25247 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.214689 25247 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.215692 25170 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.213263 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.216538 25247 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.219537 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.220853 24995 leader_election.cc:341] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.220727 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.223505 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.224174 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.225090 25104 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.228819 25145 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20240426 13:18:43.229776 25145 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.237238 24954 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.238757 25145 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.240180 25343 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.242811 25147 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.243391 25147 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.193s user 0.044s sys 0.030s
I20240426 13:18:43.249898 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.249933 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.252187 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.253333 25247 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.253901 25247 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.254215 25247 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.254511 25247 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.260174 25147 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.260936 25147 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.261242 25147 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.264398 25166 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.265038 25166 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.265427 25247 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.266041 25147 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.267696 25166 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.268460 25147 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.025s user 0.005s sys 0.003s
I20240426 13:18:43.268756 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.269142 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:43.270327 24995 leader_election.cc:341] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.270184 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.270414 25070 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.271579 25247 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.271522 24921 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:18:43.272337 25069 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.272783 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.273792 25166 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.277287 25147 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.280949 25170 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.281755 25166 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.282235 25166 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.283843 25029 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.289721 25166 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.290405 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.295082 25069 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.292549 25145 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.292769 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.296679 25147 tablet_bootstrap.cc:654] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.297076 25145 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.006s
I20240426 13:18:43.297024 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.301064 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.301245 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.300055 25145 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.304327 25102 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.307796 25145 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.308281 25145 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.309265 25148 tablet_bootstrap.cc:492] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.309927 25148 ts_tablet_manager.cc:1367] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.121s user 0.036s sys 0.045s
I20240426 13:18:43.309240 25145 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.306284 25166 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.312208 25166 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.312570 25166 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.313241 25145 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20240426 13:18:43.314337 25145 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.315069 25148 raft_consensus.cc:348] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.315747 25148 raft_consensus.cc:374] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.316002 25148 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.316589 25148 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.318992 25148 ts_tablet_manager.cc:1398] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240426 13:18:43.320574 25148 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.321081 25145 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.326591 25148 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.328107 25166 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.331084 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:43.331413 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.332201 25030 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.332669 25069 leader_election.cc:341] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.337765 25102 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.343178 25148 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.343693 25148 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.006s
I20240426 13:18:43.345180 25145 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.345890 25145 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.004s
I20240426 13:18:43.348712 25145 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.349607 25145 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.349992 25145 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.350798 25145 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.354988 25145 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20240426 13:18:43.352182 25148 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.356284 25148 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.356703 25148 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.357539 25148 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.343092 24854 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.358803 25145 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.357533 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.362203 25166 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.362831 25148 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s user 0.006s sys 0.004s
I20240426 13:18:43.364857 25388 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.365327 25388 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.366207 25388 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.365379 25148 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.363054 25030 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:43.371930 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.382025 25247 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.378944 25166 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.384305 25166 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:43.385373 25166 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.386198 25247 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.411201 25247 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.411837 25145 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.415200 24995 consensus_peers.cc:485] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.417309 25148 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.411700 25247 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.433965 24852 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:43.442292 25147 tablet_bootstrap.cc:492] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.442688 25147 ts_tablet_manager.cc:1367] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.168s user 0.037s sys 0.059s
I20240426 13:18:43.443614 25247 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.447784 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.451277 24995 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.454427 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:43.457216 25147 raft_consensus.cc:348] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.457945 25147 raft_consensus.cc:374] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.458212 25147 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.458858 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.458890 25147 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.460104 25388 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.461513 25147 ts_tablet_manager.cc:1398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20240426 13:18:43.457407 25169 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:43.463943 24996 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.464426 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.465341 25388 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.466833 25147 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.471967 25170 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.472435 25170 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.474220 25170 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.475158 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.475803 25102 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:43.476337 25149 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.481870 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:43.482648 25340 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.482991 25340 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.483301 25340 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.490619 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.491314 25029 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 0.
I20240426 13:18:43.492201 25145 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.492317 25148 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.492673 25145 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.137s user 0.015s sys 0.005s
I20240426 13:18:43.493291 25148 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.128s user 0.013s sys 0.009s
I20240426 13:18:43.494983 25145 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.495694 25148 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.496378 25148 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.495635 25145 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.496626 25148 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.496938 25145 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.503412 25148 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.506856 25148 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240426 13:18:43.512923 25148 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.526500 25145 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.535035 25145 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.042s user 0.004s sys 0.001s
I20240426 13:18:43.535688 25145 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.539978 25145 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.541980 25149 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.547727 25340 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.560426 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 57189.8us
I20240426 13:18:43.560401 25388 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.574793 25148 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.575044 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:43.582342 25340 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.582904 25220 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.569615 25388 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.584406 25220 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.600749 25220 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.600765 25388 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.626236 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:43.627065 25023 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.621696 25457 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20240426 13:18:43.635311 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.638132 25149 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:18:43.638381 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.638553 25149 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.638981 25069 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.639286 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.641469 25149 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.644457 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.645193 25147 tablet_bootstrap.cc:654] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.645061 25103 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:43.649520 25145 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.649973 25145 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.114s user 0.021s sys 0.020s
I20240426 13:18:43.652129 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.652477 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.652722 25105 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.692116 25148 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.693367 25148 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.181s user 0.016s sys 0.022s
I20240426 13:18:43.698763 25145 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.699433 25145 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.700107 25145 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.690934 25193 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.708935 25145 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.709827 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:43.694526 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58946.1us
I20240426 13:18:43.719240 25105 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:43.708782 25193 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.728883 25148 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.729614 25148 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.729880 25148 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.730433 25148 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.732911 25193 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.740386 25145 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.090s user 0.004s sys 0.016s
I20240426 13:18:43.743438 25145 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:43.740538 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.734037 25023 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:18:43.742260 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:43.748308 25148 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.055s user 0.005s sys 0.001s
I20240426 13:18:43.756583 25148 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
W20240426 13:18:43.752993 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59912us
I20240426 13:18:43.757264 25029 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:43.759464 25346 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.760121 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:18:43.756211 25070 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.762665 25346 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:18:43.768826 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.782189 25145 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:43.767239 25069 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.784906 25469 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.785822 25346 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.786357 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:43.787457 25340 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.788965 25469 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.776912 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.796224 25340 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.798624 25340 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.800470 25466 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.800793 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:43.804319 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:43.805087 25023 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:43.805917 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:43.807273 25170 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.808605 25147 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:43.809031 25147 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.342s user 0.021s sys 0.052s
I20240426 13:18:43.811282 25147 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.811950 25147 raft_consensus.cc:374] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.812340 25147 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.812939 25147 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.800117 25469 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:43.799050 24996 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:43.814049 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52370.7us
I20240426 13:18:43.817289 25147 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240426 13:18:43.826552 25147 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:43.827011 25468 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:43.829603 24852 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:43.831777 24995 leader_election.cc:341] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:43.833313 24995 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.833851 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:43.835572 25155 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:43.845559 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.849954 25466 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:43.850564 25466 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:43.851121 25466 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.855835 25148 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.865031 25170 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240426 13:18:43.871171 24995 consensus_peers.cc:485] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.884804 25147 tablet_bootstrap.cc:654] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:43.890708 25169 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:43.905288 25069 consensus_peers.cc:485] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.907903 25388 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.925091 25148 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:43.925575 25148 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.007s
I20240426 13:18:43.934667 25148 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.935341 25148 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.927335 25469 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:43.937336 24995 consensus_peers.cc:485] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:43.937630 25469 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:43.938962 25145 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:43.939409 25145 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.196s user 0.024s sys 0.022s
I20240426 13:18:43.935580 25148 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.942118 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:43.943434 25148 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.945951 25148 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20240426 13:18:43.947440 25148 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:43.953025 25145 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.952803 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:43.954293 25145 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:43.954643 25104 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:43.959250 25145 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:43.962679 25104 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:43.959928 25145 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.964603 25220 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.968847 25145 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20240426 13:18:43.970192 25145 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
W20240426 13:18:43.966300 24995 leader_election.cc:341] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:43.951889 25466 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.979049 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:43.980113 25469 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:43.980737 25469 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:43.983258 25169 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:43.985306 25476 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:43.985920 25476 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:43.988385 25476 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:43.981571 25469 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.025846 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.026504 25025 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.033120 25145 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.039769 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.044103 25148 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.046036 25147 tablet_bootstrap.cc:492] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.046413 25147 ts_tablet_manager.cc:1367] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.221s user 0.013s sys 0.006s
I20240426 13:18:44.052289 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:44.055150 25476 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.055516 25476 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.056268 25476 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.057085 25147 raft_consensus.cc:348] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.057709 25147 raft_consensus.cc:374] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.057911 25147 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.058398 25147 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.058954 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.066107 25466 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.070221 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.078383 25023 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.074592 25476 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.075081 24853 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:44.061372 25069 leader_election.cc:341] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.085214 25069 leader_election.cc:341] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.088146 24956 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.095115 25148 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.095729 25148 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.000s
I20240426 13:18:44.099848 25469 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.100349 25469 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.077260 25147 ts_tablet_manager.cc:1398] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.031s user 0.003s sys 0.002s
I20240426 13:18:44.101970 25147 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.103034 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.104373 24995 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.105665 25469 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.108811 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.109436 25145 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.109469 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.110006 25145 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.140s user 0.008s sys 0.001s
I20240426 13:18:44.109854 25148 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.110812 25148 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.091506 25023 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.111435 25148 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.112100 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.112848 25220 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.112630 25148 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.113301 25220 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.114078 25220 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.114902 25148 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240426 13:18:44.115773 25148 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.116256 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.117322 25476 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.117911 25388 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.118274 25388 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.118583 25388 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.120406 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.120968 24954 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.126757 24954 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.127559 25148 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.128356 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.128439 25343 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:18:44.132144 25023 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.128659 25388 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.132305 25343 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.135627 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.135892 25343 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.136283 25104 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.138972 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.139794 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.140642 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:18:44.141271 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.141433 25149 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.141892 25149 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.142803 25346 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:44.143616 25149 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.144125 25169 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.144245 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.144623 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.147641 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.145946 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.147612 25169 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.148761 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:44.149644 25149 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.146441 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.149955 25149 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.149921 25169 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.150496 25149 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.150336 25104 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.151762 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:44.152599 25469 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.153085 25469 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:44.153349 25145 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.153849 25469 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.154603 25069 leader_election.cc:341] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.156533 24995 leader_election.cc:341] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.156461 25149 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.153993 25145 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.158223 25145 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.158839 25145 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.159154 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:44.159684 25025 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.160766 25388 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.162973 25147 tablet_bootstrap.cc:654] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.166399 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:44.167341 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:44.167341 25145 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20240426 13:18:44.168090 25220 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.172191 25145 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.172272 25220 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:44.173105 25220 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.176745 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.178095 25069 leader_election.cc:341] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.168404 25149 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.179697 25145 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.181077 25148 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.181524 25148 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.001s
I20240426 13:18:44.184593 25343 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.179822 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.197070 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:44.199381 25148 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.202534 25148 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.202804 25148 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.203717 25148 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.206888 25148 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20240426 13:18:44.207747 25148 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.207835 25147 tablet_bootstrap.cc:492] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.208339 25147 ts_tablet_manager.cc:1367] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.005s
I20240426 13:18:44.209322 25346 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.210690 25147 raft_consensus.cc:348] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.211360 25147 raft_consensus.cc:374] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.211614 25147 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.212205 25147 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.214160 25147 ts_tablet_manager.cc:1398] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240426 13:18:44.215096 25147 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.214197 25148 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.220003 25147 tablet_bootstrap.cc:654] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.224470 24853 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.226548 24853 catalog_manager.cc:5484] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:44.233242 25147 tablet_bootstrap.cc:492] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.231274 25148 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.231062 25145 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.235157 25145 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.010s
I20240426 13:18:44.237413 25145 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.238093 25145 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.238333 25145 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.239094 25145 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.241173 25145 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:44.241990 25145 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.244344 25148 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.001s
I20240426 13:18:44.245644 25343 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.246137 25343 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.247035 25145 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.247967 25343 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.247848 25148 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.248560 25148 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.248833 25148 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.249415 25148 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.248337 25147 ts_tablet_manager.cc:1367] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s
I20240426 13:18:44.252068 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.251014 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.268060 25147 raft_consensus.cc:348] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.268769 25147 raft_consensus.cc:374] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.269038 25147 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.270098 25147 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.271641 25147 ts_tablet_manager.cc:1398] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240426 13:18:44.271862 24954 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.272507 25147 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.251505 25148 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240426 13:18:44.277019 25148 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.279157 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.294334 25148 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.300602 25145 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.301064 25145 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.008s
I20240426 13:18:44.301323 25147 tablet_bootstrap.cc:654] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.303562 25145 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.304288 25145 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.304582 25145 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.304756 25148 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.305222 25148 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.011s
I20240426 13:18:44.305212 25145 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.307065 25145 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240426 13:18:44.307636 25148 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.308163 25147 tablet_bootstrap.cc:492] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.308326 25148 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.308452 25460 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.308568 25147 ts_tablet_manager.cc:1367] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.001s
I20240426 13:18:44.308604 25148 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.309435 25148 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.310611 25145 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.310945 25147 raft_consensus.cc:348] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.311545 25147 raft_consensus.cc:374] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.311789 25147 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.312345 25147 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.314575 25147 ts_tablet_manager.cc:1398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240426 13:18:44.315714 25147 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.316468 25148 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20240426 13:18:44.317289 25148 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.320544 25147 tablet_bootstrap.cc:654] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.322471 25148 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.322767 25145 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.348591 25147 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.349174 25147 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.012s
I20240426 13:18:44.352315 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.352171 25147 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.353137 25147 raft_consensus.cc:374] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.353478 25147 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.354224 25147 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.356961 25147 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20240426 13:18:44.358034 25147 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.362214 25340 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.362672 25340 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.364814 25147 tablet_bootstrap.cc:654] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.365101 25340 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.366554 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.367282 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.368048 25025 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:44.372480 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.375886 25460 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.377494 25147 tablet_bootstrap.cc:492] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.378094 25147 ts_tablet_manager.cc:1367] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20240426 13:18:44.380738 25147 raft_consensus.cc:348] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.381400 25147 raft_consensus.cc:374] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.381672 25147 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.382498 25147 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.384907 25147 ts_tablet_manager.cc:1398] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20240426 13:18:44.385726 25147 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.386480 25247 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.387248 25460 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.386950 25247 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.391177 25247 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.392395 25147 tablet_bootstrap.cc:654] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.392915 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.393436 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.393947 25104 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.394971 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:18:44.395641 24995 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.396173 25469 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.396507 25469 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.396792 25469 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.397400 25169 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.397769 25169 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.400159 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.400929 25169 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.400882 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.402341 25069 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.407075 25469 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.410466 25025 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:44.413666 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.415006 25169 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.416196 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.417464 24995 leader_election.cc:341] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.418514 25469 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.419503 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.423151 25169 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.428143 25145 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.428561 25145 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.118s user 0.021s sys 0.014s
I20240426 13:18:44.432202 25469 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.432168 25104 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.432879 25469 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.434264 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.435096 25247 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.435469 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.435706 25247 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:18:44.437043 24995 leader_election.cc:341] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.436859 25247 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.439527 25469 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.451236 25469 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.453503 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.456022 25148 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.456890 25148 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.140s user 0.023s sys 0.050s
I20240426 13:18:44.462445 25145 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.463135 25145 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.463384 25145 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.463987 25145 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.454190 25104 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.466854 25460 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.467262 25460 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.469944 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.472188 25460 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.472388 25148 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.473183 25145 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20240426 13:18:44.473524 25147 tablet_bootstrap.cc:492] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.475180 25148 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.476022 25148 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.476377 25145 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.476657 25148 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.477314 25388 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.477694 25388 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.478641 25147 ts_tablet_manager.cc:1367] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.093s user 0.041s sys 0.044s
I20240426 13:18:44.478717 25148 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240426 13:18:44.479570 25148 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Bootstrap starting.
I20240426 13:18:44.480199 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.480998 25147 raft_consensus.cc:348] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.482127 25145 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.482710 25147 raft_consensus.cc:374] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.477988 24954 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.483150 25147 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.481493 25388 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:44.482385 24995 leader_election.cc:341] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.486804 25147 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.488163 24854 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:44.491271 25247 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.485638 25148 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.493719 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.494055 25147 ts_tablet_manager.cc:1398] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240426 13:18:44.494534 25247 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.494868 25247 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.494932 25147 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.495175 25247 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.497043 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.497666 25104 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.503594 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.504225 25104 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:18:44.507988 24921 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:44.509111 25340 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.509907 25147 tablet_bootstrap.cc:654] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.510025 24996 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.512785 25469 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.511269 25247 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.517969 25247 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.528340 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:44.513149 25469 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:18:44.531675 24995 leader_election.cc:341] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.536207 25469 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.538102 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.541034 25102 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.541072 25340 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.542028 25247 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.542516 25247 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.546038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.547211 25247 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:44.547359 24995 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.548976 25169 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.549651 25102 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:18:44.551628 25169 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.557210 25220 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:44.594019 25388 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.594535 25388 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.596738 25388 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.598625 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:44.600145 24995 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.600597 25145 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
I20240426 13:18:44.601087 25145 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.125s user 0.013s sys 0.002s
W20240426 13:18:44.605439 25069 consensus_peers.cc:485] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:44.612424 25476 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20240426 13:18:44.649086 25148 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: No bootstrap required, opened a new log
I20240426 13:18:44.649574 25148 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent bootstrapping tablet: real 0.170s user 0.008s sys 0.005s
I20240426 13:18:44.653059 25147 tablet_bootstrap.cc:492] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:44.653563 25147 ts_tablet_manager.cc:1367] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.159s user 0.029s sys 0.013s
I20240426 13:18:44.655938 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.656782 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 0.
I20240426 13:18:44.668563 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.678723 25147 raft_consensus.cc:348] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.679292 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:44.679510 25147 raft_consensus.cc:374] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.679831 25147 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.680466 25147 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.681375 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:44.682727 25510 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:44.682986 25148 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.683569 25509 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.684077 25509 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.684427 25509 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.683810 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:44.683794 25148 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.686348 25148 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.691991 25145 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.692965 25145 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:44.693291 25145 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:44.687101 25148 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.698750 25147 ts_tablet_manager.cc:1398] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20240426 13:18:44.699934 25147 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:44.696195 25145 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.706163 25148 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent starting tablet: real 0.056s user 0.005s sys 0.001s
I20240426 13:18:44.706499 25469 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.707794 24901 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240426 13:18:44.708371 25469 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.712450 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:44.714418 24995 leader_election.cc:341] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.717689 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:44.729259 25169 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240426 13:18:44.702035 24996 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.729920 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:44.731423 25388 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:44.749344 25169 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.749917 25145 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.149s user 0.016s sys 0.002s
I20240426 13:18:44.749879 25169 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.754920 25149 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.755331 25149 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.756888 25149 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.758384 25169 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.758152 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.761008 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.762372 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:44.763312 25169 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.763725 25169 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.764123 25169 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.768628 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.752341 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:44.790817 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:44.806080 25145 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Bootstrap starting.
I20240426 13:18:44.807621 25169 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.817451 25169 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:44.810760 25510 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:44.816080 25103 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:44.816162 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:44.859699 25510 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:44.889567 25516 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240426 13:18:44.891706 25147 tablet_bootstrap.cc:654] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:44.876242 25027 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:44.923633 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:18:44.937918 25069 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:44.939030 25069 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:44.941195 25519 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:44.941871 25519 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:44.944573 25476 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:44.989790 25476 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:44.990069 25519 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:44.985904 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.015215 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:45.017508 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76585.4us
I20240426 13:18:45.015992 25476 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.059618 25145 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Neither blocks nor log segments found. Creating new log.
W20240426 13:18:45.067466 25521 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240426 13:18:45.089540 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.063863 25509 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.114344 25509 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.091749 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:18:45.154125 25069 leader_election.cc:341] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.143637 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:18:45.129640 25023 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.178337 25103 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
W20240426 13:18:45.180536 24996 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:45.207374 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45983.7us
I20240426 13:18:45.237300 25147 tablet_bootstrap.cc:492] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.237917 25147 ts_tablet_manager.cc:1367] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.538s user 0.045s sys 0.053s
I20240426 13:18:45.277374 25145 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: No bootstrap required, opened a new log
W20240426 13:18:45.268664 24995 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.277824 25145 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent bootstrapping tablet: real 0.472s user 0.017s sys 0.000s
I20240426 13:18:45.277981 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:18:45.280141 25145 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.280794 25145 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.281035 25145 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.281570 25145 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.282732 25145 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240426 13:18:45.282754 25023 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.256166 25476 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.299422 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:18:45.306221 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58679us
I20240426 13:18:45.305428 25147 raft_consensus.cc:348] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.307574 25147 raft_consensus.cc:374] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.307849 25147 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.298511 25531 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.308450 25147 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.299913 25388 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.321305 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:45.321309 25536 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.269781 24996 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.325333 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.325436 25476 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.306404 25533 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.330307 25147 ts_tablet_manager.cc:1398] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.092s user 0.008s sys 0.000s
I20240426 13:18:45.333439 25147 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.336850 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:45.337469 25028 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.323012 25536 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.341638 25536 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.349905 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:18:45.357246 25520 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.303s user 0.004s sys 0.000s
I20240426 13:18:45.334803 25533 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.330426 25531 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.360585 24954 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
W20240426 13:18:45.349844 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85093.6us
I20240426 13:18:45.362109 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.362440 25531 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.362920 25025 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 0.
I20240426 13:18:45.360642 25533 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.369011 25544 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.369866 25544 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:45.371042 25544 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.397341 24854 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.401340 25343 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:18:45.401881 25343 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.404888 25028 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.406941 25343 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:18:45.409471 24995 consensus_peers.cc:485] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:18:45.412189 25069 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
W20240426 13:18:45.413578 24995 leader_election.cc:341] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.418368 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.423952 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.425513 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.426826 25476 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240426 13:18:45.427258 25476 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:18:45.427618 25476 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.432884 25147 tablet_bootstrap.cc:654] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.435739 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.436437 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.436904 25476 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.437959 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:18:45.438534 25100 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.438735 25476 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:45.441107 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:18:45.442044 25525 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.442466 25525 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.442476 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:18:45.443267 25525 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:45.443753 25069 leader_election.cc:341] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.443715 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:18:45.445371 25069 leader_election.cc:341] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Illegal state: must be running to vote when last-logged opid is not known
I20240426 13:18:45.445989 25100 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
W20240426 13:18:45.460340 24995 consensus_peers.cc:485] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Couldn't send request to peer 1e87e7bb414642aeb72fe0d2ff72cc2e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:45.471143 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.473078 25147 tablet_bootstrap.cc:492] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.473265 25247 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.473645 25147 ts_tablet_manager.cc:1367] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.006s
I20240426 13:18:45.473704 25247 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:18:45.473960 25538 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:45.474552 25247 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.475997 25147 raft_consensus.cc:348] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.476691 25147 raft_consensus.cc:374] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.476928 25147 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.477502 25147 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.480017 25147 ts_tablet_manager.cc:1398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240426 13:18:45.484851 25147 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.489279 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.490435 25147 tablet_bootstrap.cc:654] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.493149 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:18:45.493873 25025 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.508806 25340 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.512437 25343 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.513057 25343 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.514863 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.515576 25343 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.522156 25340 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:18:45.503795 25025 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:18:45.530844 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:18:45.531678 25476 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 13:18:45.532244 25476 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:18:45.533039 25476 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.536098 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.537350 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.551002 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.552122 25343 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.579582 25147 tablet_bootstrap.cc:492] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.579937 25147 ts_tablet_manager.cc:1367] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.095s user 0.037s sys 0.048s
I20240426 13:18:45.581733 25147 raft_consensus.cc:348] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.582262 25147 raft_consensus.cc:374] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.582463 25147 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.582959 25147 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.584800 25147 ts_tablet_manager.cc:1398] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240426 13:18:45.585673 25147 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.590191 25147 tablet_bootstrap.cc:654] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.589643 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.591845 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.599486 25147 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.599920 25147 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240426 13:18:45.602066 25147 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.602736 25147 raft_consensus.cc:374] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.602974 25147 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.603569 25147 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.607304 25147 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240426 13:18:45.608093 25147 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.610599 24854 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.613314 25147 tablet_bootstrap.cc:654] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.613054 24854 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:45.618335 25147 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.618755 25147 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20240426 13:18:45.620955 25147 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.621559 25147 raft_consensus.cc:374] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.621827 25147 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.622390 25147 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.624413 25147 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20240426 13:18:45.625187 25147 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.630129 25147 tablet_bootstrap.cc:654] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.633138 24954 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.636466 25247 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.639036 24852 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:18:45.685688 25147 tablet_bootstrap.cc:492] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.686152 25147 ts_tablet_manager.cc:1367] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.061s user 0.025s sys 0.032s
I20240426 13:18:45.688338 25147 raft_consensus.cc:348] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.688982 25147 raft_consensus.cc:374] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.689254 25147 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.689792 25147 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.691872 25147 ts_tablet_manager.cc:1398] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20240426 13:18:45.692725 25147 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.696981 25147 tablet_bootstrap.cc:654] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.701676 25147 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.702126 25147 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240426 13:18:45.704283 25147 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.704733 25147 raft_consensus.cc:374] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.704933 25147 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.705405 25147 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.706854 25147 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240426 13:18:45.707468 25147 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Bootstrap starting.
I20240426 13:18:45.712188 25147 tablet_bootstrap.cc:654] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Neither blocks nor log segments found. Creating new log.
I20240426 13:18:45.723409 25147 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: No bootstrap required, opened a new log
I20240426 13:18:45.723860 25147 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240426 13:18:45.726006 25147 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.726652 25147 raft_consensus.cc:374] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 13:18:45.726905 25147 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Initialized, Role: FOLLOWER
I20240426 13:18:45.727519 25147 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.729601 25147 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240426 13:18:45.763857 25531 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.764367 25531 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.766121 25531 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.767045 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.767042 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.769428 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.770242 25531 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.807253 24948 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.813416 25528 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.823168 25247 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.826627 24954 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.841818 25531 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.842206 25531 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:18:45.843596 25531 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.844229 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.844869 25028 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:45.844631 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.846194 24921 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.847136 25531 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.852551 25531 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.855785 25525 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.865020 24954 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.869930 25533 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.876541 25533 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.882001 25533 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240426 13:18:45.919626 24954 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.926949 25247 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.930404 23796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 13:18:45.942155 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.944239 25343 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:18:45.944952 25343 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:18:45.945730 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.948276 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:18:45.949141 25025 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 1: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:18:45.951996 25343 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:45.953299 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:18:45.957528 24921 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:18:45.957765 25544 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:45.958447 25531 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 0 FOLLOWER]: Advancing to term 1
I20240426 13:18:45.979741 23796 server_base.cc:1034] running on GCE node
W20240426 13:18:45.980520 25581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 13:18:45.982230 25531 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240426 13:18:45.983752 23796 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 13:18:45.984112 23796 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 13:18:45.984354 23796 hybrid_clock.cc:648] HybridClock initialized: now 1714137525984338 us; error 0 us; skew 500 ppm
I20240426 13:18:45.985146 23796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 13:18:45.992588 23796 webserver.cc:460] Webserver started at http://127.23.61.4:46465/ using document root <none> and password file <none>
I20240426 13:18:45.993144 23796 fs_manager.cc:359] Metadata directory not provided
I20240426 13:18:45.993309 23796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 13:18:45.993568 23796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 13:18:45.994710 23796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/instance:
uuid: "1590f80bdade41c6a755b8501c55fe16"
format_stamp: "Formatted at 2024-04-26 13:18:45 on dist-test-slave-jx77"
I20240426 13:18:45.999164 23796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240426 13:18:46.002399 25586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:46.003178 23796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240426 13:18:46.003441 23796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
uuid: "1590f80bdade41c6a755b8501c55fe16"
format_stamp: "Formatted at 2024-04-26 13:18:45 on dist-test-slave-jx77"
I20240426 13:18:46.003692 23796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 13:18:46.024974 23796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240426 13:18:46.026360 23796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 13:18:46.027802 23796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 13:18:46.030392 23796 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 13:18:46.030674 23796 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:46.030978 23796 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 13:18:46.031191 23796 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240426 13:18:46.113039 25525 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:46.133138 25525 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240426 13:18:46.170192 23796 rpc_server.cc:307] RPC server started. Bound to: 127.23.61.4:40069
I20240426 13:18:46.170292 25650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.61.4:40069 every 8 connection(s)
I20240426 13:18:46.205528 23796 auto_leader_rebalancer-test.cc:405] add a tserver: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:46.214516 25651 heartbeater.cc:347] Connected to a master server at 127.23.61.62:34153
I20240426 13:18:46.214915 25651 heartbeater.cc:464] Registering TS with master...
I20240426 13:18:46.215574 25651 heartbeater.cc:511] Master 127.23.61.62:34153 requested a full tablet report, sending...
I20240426 13:18:46.217203 24851 ts_manager.cc:194] Registered new tserver with Master: 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:18:46.218364 24851 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60836
I20240426 13:18:46.826489 25100 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:46.833271 25025 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.834826 24953 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.835011 25655 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.836373 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.842222 25525 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.843525 24953 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.851163 25025 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:18:46.851903 25069 consensus_peers.cc:485] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e915e90a1a6847899575125ef7e68f2a. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:46.866313 24851 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.876639 24953 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:46.885234 25100 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.885540 25025 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:46.887223 25531 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:46.887972 25340 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:46.907922 24920 consensus_peers.cc:485] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4076b67a4fe4b3297158d094e10de08. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:46.920836 25340 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.922333 25028 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:46.938758 24952 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.013811 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:47.016237 24920 consensus_peers.cc:485] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f0f51d042f24e54b9e3a33e390f5982. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:47.018324 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.020339 25670 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:47.019953 25100 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.022071 25027 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.029992 25672 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.043183 25670 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.044119 25104 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.052625 25027 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.064329 24854 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.075425 25104 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.081804 25031 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:47.083072 25655 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.088611 24952 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:47.088995 25069 consensus_peers.cc:485] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a28ef3032ef44d2d89c750a8d4892f00. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:47.090783 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:47.100252 25530 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.103767 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.115046 25031 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:47.117615 24851 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:47.220291 25651 heartbeater.cc:503] Master 127.23.61.62:34153 was elected leader, sending a full tablet report...
I20240426 13:18:47.327930 25678 ts_tablet_manager.cc:902] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:47.332517 25678 tablet_copy_client.cc:308] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:47.356379 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet e915e90a1a6847899575125ef7e68f2a from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:47.357034 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet e915e90a1a6847899575125ef7e68f2a from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a
I20240426 13:18:47.363461 25115 tablet_copy_source_session.cc:205] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.367094 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e915e90a1a6847899575125ef7e68f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.378876 25678 tablet_copy_client.cc:783] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.379418 25678 tablet_copy_client.cc:647] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.383457 25678 tablet_copy_client.cc:523] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.391124 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.408108 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.408793 25678 tablet_bootstrap.cc:492] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.409396 25678 ts_tablet_manager.cc:1367] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240426 13:18:47.411641 25678 raft_consensus.cc:348] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.412413 25678 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.413005 25678 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.415668 25678 ts_tablet_manager.cc:1398] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240426 13:18:47.417117 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:47.417500 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-e915e90a1a6847899575125ef7e68f2a on tablet e915e90a1a6847899575125ef7e68f2a with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.420979 25678 ts_tablet_manager.cc:902] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:47.432463 25678 tablet_copy_client.cc:308] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:47.445921 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 8f0f51d042f24e54b9e3a33e390f5982 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:47.446491 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 8f0f51d042f24e54b9e3a33e390f5982 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982
I20240426 13:18:47.453779 24966 tablet_copy_source_session.cc:205] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.456410 25678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f51d042f24e54b9e3a33e390f5982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.466979 25678 tablet_copy_client.cc:783] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.467446 25678 tablet_copy_client.cc:647] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.472317 25687 ts_tablet_manager.cc:902] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:47.475312 25678 tablet_copy_client.cc:523] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.482676 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.488080 25687 tablet_copy_client.cc:308] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:47.489789 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet f4076b67a4fe4b3297158d094e10de08 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:47.490343 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet f4076b67a4fe4b3297158d094e10de08 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08
I20240426 13:18:47.497515 24966 tablet_copy_source_session.cc:205] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.500232 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4076b67a4fe4b3297158d094e10de08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.506209 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.506920 25678 tablet_bootstrap.cc:492] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.507458 25678 ts_tablet_manager.cc:1367] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.005s
I20240426 13:18:47.510426 25678 raft_consensus.cc:348] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.511133 25678 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.511603 25678 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.516321 25687 tablet_copy_client.cc:783] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.516829 25687 tablet_copy_client.cc:647] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.519228 25678 ts_tablet_manager.cc:1398] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20240426 13:18:47.522756 25687 tablet_copy_client.cc:523] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.525921 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:47.526297 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-8f0f51d042f24e54b9e3a33e390f5982 on tablet 8f0f51d042f24e54b9e3a33e390f5982 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.533028 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.555169 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.556638 25687 tablet_bootstrap.cc:492] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.557236 25687 ts_tablet_manager.cc:1367] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.006s
I20240426 13:18:47.559645 25687 raft_consensus.cc:348] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.560464 25687 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.561110 25687 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.563676 25687 ts_tablet_manager.cc:1398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240426 13:18:47.565120 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:47.566191 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f4076b67a4fe4b3297158d094e10de08 on tablet f4076b67a4fe4b3297158d094e10de08 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.607456 25687 ts_tablet_manager.cc:902] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:47.610935 25687 tablet_copy_client.cc:308] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:47.612406 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet a28ef3032ef44d2d89c750a8d4892f00 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:47.612838 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet a28ef3032ef44d2d89c750a8d4892f00 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00
I20240426 13:18:47.618269 25115 tablet_copy_source_session.cc:205] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:47.622820 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28ef3032ef44d2d89c750a8d4892f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:47.636518 25687 tablet_copy_client.cc:783] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:47.637032 25687 tablet_copy_client.cc:647] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:47.642630 25687 tablet_copy_client.cc:523] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:47.650103 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:47.666124 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:47.666869 25687 tablet_bootstrap.cc:492] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:47.667435 25687 ts_tablet_manager.cc:1367] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240426 13:18:47.669865 25687 raft_consensus.cc:348] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.670625 25687 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:47.671721 25687 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:47.673887 25687 ts_tablet_manager.cc:1398] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240426 13:18:47.675633 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:47.675983 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-a28ef3032ef44d2d89c750a8d4892f00 on tablet a28ef3032ef44d2d89c750a8d4892f00 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:47.705282 25624 raft_consensus.cc:1203] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:47.816872 25624 raft_consensus.cc:1203] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:47.923321 25624 raft_consensus.cc:1203] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:48.021375 25624 raft_consensus.cc:1203] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:48.229637 25672 raft_consensus.cc:1050] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.231947 25672 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.237628 25624 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.238708 25706 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.240613 25104 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.241484 25031 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.242543 25706 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.244088 25672 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.249917 25666 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.256520 25031 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.259442 25624 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.269929 25099 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.275172 24851 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.301064 24948 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.305279 25624 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.305763 25100 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.306443 25666 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.308753 25666 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.312630 25706 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.314966 25705 raft_consensus.cc:1050] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.318531 25099 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.319116 25705 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.321382 25672 raft_consensus.cc:1050] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.323690 25672 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.326419 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f0f51d042f24e54b9e3a33e390f5982 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.326228 25624 raft_consensus.cc:2932] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.341295 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.352078 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.353654 25533 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.359977 25031 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.361769 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.365299 24953 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.366580 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.378365 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0f51d042f24e54b9e3a33e390f5982 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:48.390395 25624 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.392375 25672 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:48.393044 25533 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.397210 25028 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.399066 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.402194 25028 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.420107 25099 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.421591 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.423954 25723 tablet_replica.cc:330] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:48.427280 25666 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.432864 25027 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.434691 25624 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.441321 24953 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.450798 25723 raft_consensus.cc:2229] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.451895 25723 raft_consensus.cc:2258] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.453533 24852 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.457679 25624 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.476174 25723 ts_tablet_manager.cc:1875] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.486044 24854 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:48.487012 25104 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.502538 25723 ts_tablet_manager.cc:1888] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.503041 25723 log.cc:1196] T 8f0f51d042f24e54b9e3a33e390f5982 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/8f0f51d042f24e54b9e3a33e390f5982
I20240426 13:18:48.505415 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 8f0f51d042f24e54b9e3a33e390f5982 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.508299 25104 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.512339 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.513257 25031 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.513634 25705 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.514560 25530 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.521955 25530 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.526067 24953 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.530419 25031 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.532763 25623 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.533866 25533 raft_consensus.cc:1050] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:48.535830 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.536146 25533 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.537492 25028 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.541508 25711 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.546309 25622 raft_consensus.cc:2932] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.552376 25666 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.565212 24854 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.567903 25622 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.591521 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4076b67a4fe4b3297158d094e10de08 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.603896 25027 raft_consensus.cc:2932] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.608283 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.630558 25031 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.632150 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.634680 24953 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.636488 25705 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.638643 25622 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:48.640993 25722 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:48.654731 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet e915e90a1a6847899575125ef7e68f2a with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:48.658466 25734 tablet_replica.cc:330] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:48.670106 25734 raft_consensus.cc:2229] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.670925 25734 raft_consensus.cc:2258] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.674072 25734 ts_tablet_manager.cc:1875] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.706372 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e915e90a1a6847899575125ef7e68f2a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.743126 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.754417 25533 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.765748 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet f4076b67a4fe4b3297158d094e10de08 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:48.771764 25027 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:18:48.805292 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48469.7us
I20240426 13:18:48.813553 25622 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.820850 25737 tablet_replica.cc:330] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:48.821174 24851 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.827250 25734 ts_tablet_manager.cc:1888] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.827991 25734 log.cc:1196] T e915e90a1a6847899575125ef7e68f2a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/e915e90a1a6847899575125ef7e68f2a
I20240426 13:18:48.830166 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet e915e90a1a6847899575125ef7e68f2a (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.857878 25737 raft_consensus.cc:2229] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.863714 25737 raft_consensus.cc:2258] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.873905 25104 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:48.878955 25737 ts_tablet_manager.cc:1875] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.880198 25622 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.880179 24948 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:48.882046 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.889048 25533 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:48.891849 24948 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.893781 25737 ts_tablet_manager.cc:1888] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.894222 25737 log.cc:1196] T f4076b67a4fe4b3297158d094e10de08 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/f4076b67a4fe4b3297158d094e10de08
I20240426 13:18:48.895793 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet f4076b67a4fe4b3297158d094e10de08 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:48.890522 25530 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.899255 25622 raft_consensus.cc:2932] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.908447 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a28ef3032ef44d2d89c750a8d4892f00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:48.936909 24852 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:48.954304 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet a28ef3032ef44d2d89c750a8d4892f00 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:48.955641 25723 tablet_replica.cc:330] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:48.973716 25723 raft_consensus.cc:2229] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:48.974592 25723 raft_consensus.cc:2258] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:48.979606 25723 ts_tablet_manager.cc:1875] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:48.992491 25723 ts_tablet_manager.cc:1888] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:48.992846 25723 log.cc:1196] T a28ef3032ef44d2d89c750a8d4892f00 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/a28ef3032ef44d2d89c750a8d4892f00
I20240426 13:18:48.994138 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet a28ef3032ef44d2d89c750a8d4892f00 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:50.006062 25104 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.013796 24948 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.014156 25027 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:50.014070 25069 consensus_peers.cc:485] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4680c139c46f44b49096bfd65f033a32. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.015159 25739 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.015949 25533 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.024395 24948 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.030057 25027 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.022917 25533 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.039239 24854 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.074087 24948 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:50.092530 24920 consensus_peers.cc:485] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 786abf3f43724ad5b959e497fc776d1d. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.092675 25104 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.094115 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.097270 25743 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.099260 25666 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:50.114991 25744 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.129842 25105 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.135630 24953 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:50.150375 24920 consensus_peers.cc:485] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3a669ffe77a4ad29604f4ecd7c2644a. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.147704 25031 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.146884 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.157034 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.158200 25105 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.159240 25666 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.160166 25743 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.175819 25743 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.186203 25105 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.193562 25031 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.197342 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.204331 25028 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.222267 25105 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.222569 24953 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:50.224629 25751 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:50.225574 25247 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:50.233036 24995 consensus_peers.cc:485] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a2ec72789da4a7383ca5ce0f90a4563. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:50.234961 25751 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.236155 24948 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.238086 25105 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:50.252436 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:50.574813 25762 ts_tablet_manager.cc:902] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:50.608137 25762 tablet_copy_client.cc:308] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:50.609663 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 4680c139c46f44b49096bfd65f033a32 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:50.610153 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 4680c139c46f44b49096bfd65f033a32 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32
I20240426 13:18:50.616380 25115 tablet_copy_source_session.cc:205] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.619112 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4680c139c46f44b49096bfd65f033a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.629402 25767 ts_tablet_manager.cc:902] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:50.639420 25762 tablet_copy_client.cc:783] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.640480 25762 tablet_copy_client.cc:647] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.644645 25767 tablet_copy_client.cc:308] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:50.647989 25762 tablet_copy_client.cc:523] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.651793 25773 ts_tablet_manager.cc:902] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:50.658587 25773 tablet_copy_client.cc:308] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:50.662590 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 786abf3f43724ad5b959e497fc776d1d from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:50.663107 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 786abf3f43724ad5b959e497fc776d1d from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d
I20240426 13:18:50.663457 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.665414 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 8a2ec72789da4a7383ca5ce0f90a4563 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:50.665997 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 8a2ec72789da4a7383ca5ce0f90a4563 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563
I20240426 13:18:50.674685 24966 tablet_copy_source_session.cc:205] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.677501 25041 tablet_copy_source_session.cc:205] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.679965 25767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ec72789da4a7383ca5ce0f90a4563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.681298 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786abf3f43724ad5b959e497fc776d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.714887 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.716423 25762 tablet_bootstrap.cc:492] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.717061 25762 ts_tablet_manager.cc:1367] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.054s user 0.028s sys 0.011s
I20240426 13:18:50.719095 25773 tablet_copy_client.cc:783] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.720983 25779 ts_tablet_manager.cc:902] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:50.719741 25762 raft_consensus.cc:348] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.723388 25762 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.723858 25767 tablet_copy_client.cc:783] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.724459 25767 tablet_copy_client.cc:647] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.724964 25773 tablet_copy_client.cc:647] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.724066 25762 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.729677 25767 tablet_copy_client.cc:523] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.732482 25779 tablet_copy_client.cc:308] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:50.736989 24965 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet d3a669ffe77a4ad29604f4ecd7c2644a from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:50.737527 24965 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet d3a669ffe77a4ad29604f4ecd7c2644a from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a
I20240426 13:18:50.737974 25762 ts_tablet_manager.cc:1398] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.021s user 0.010s sys 0.001s
I20240426 13:18:50.741219 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:50.741624 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-4680c139c46f44b49096bfd65f033a32 on tablet 4680c139c46f44b49096bfd65f033a32 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.746167 24965 tablet_copy_source_session.cc:205] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:50.750715 25773 tablet_copy_client.cc:523] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.755739 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.757926 25779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a669ffe77a4ad29604f4ecd7c2644a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:50.762075 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.782711 25779 tablet_copy_client.cc:783] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:50.783242 25779 tablet_copy_client.cc:647] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:50.786840 25779 tablet_copy_client.cc:523] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:50.793212 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.794307 25773 tablet_bootstrap.cc:492] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.794966 25773 ts_tablet_manager.cc:1367] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.007s
I20240426 13:18:50.798077 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.798283 25773 raft_consensus.cc:348] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.798990 25767 tablet_bootstrap.cc:492] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.799268 25773 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.799748 25767 ts_tablet_manager.cc:1367] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.007s
I20240426 13:18:50.799953 25773 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.802019 25773 ts_tablet_manager.cc:1398] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240426 13:18:50.802492 25767 raft_consensus.cc:348] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.803373 25767 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.803800 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:50.804229 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-786abf3f43724ad5b959e497fc776d1d on tablet 786abf3f43724ad5b959e497fc776d1d with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.803951 25767 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.805538 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:50.806308 25767 ts_tablet_manager.cc:1398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240426 13:18:50.807772 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:50.808197 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-8a2ec72789da4a7383ca5ce0f90a4563 on tablet 8a2ec72789da4a7383ca5ce0f90a4563 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:50.831256 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:50.832229 25779 tablet_bootstrap.cc:492] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:50.832898 25779 ts_tablet_manager.cc:1367] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.004s
I20240426 13:18:50.835075 25779 raft_consensus.cc:348] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.835621 25779 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:50.836069 25779 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:50.838259 25779 ts_tablet_manager.cc:1398] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240426 13:18:50.839499 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:50.839851 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-d3a669ffe77a4ad29604f4ecd7c2644a on tablet d3a669ffe77a4ad29604f4ecd7c2644a with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:51.069008 25622 raft_consensus.cc:1203] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:51.199012 25622 raft_consensus.cc:1203] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:51.226763 25622 raft_consensus.cc:1203] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:51.240890 25622 raft_consensus.cc:1203] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:51.592974 25789 raft_consensus.cc:1050] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.595327 25789 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.600955 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.601598 25622 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.602257 25789 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 13:18:51.608490 24948 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.608378 25751 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.611637 25792 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:51.635389 25751 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.637683 24948 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.640980 25622 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.652123 24852 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.660604 25790 raft_consensus.cc:1050] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.663007 25790 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.659673 25099 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.680591 25622 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.682744 25790 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.688356 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.688584 25099 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.691300 25790 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:51.692549 25744 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.710956 25801 raft_consensus.cc:1050] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.713357 25801 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.724493 25027 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.726487 24948 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.728202 25800 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.728871 25622 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.730080 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.732156 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.736451 25744 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.742054 25027 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.739856 25099 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.796662 25028 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.800803 25622 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.804611 25789 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.807399 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.808974 25792 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.814325 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.816243 25104 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.817382 25739 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.822872 25623 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.828912 25624 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.833739 25622 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.814905 25247 raft_consensus.cc:2932] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.852490 24854 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:51.862551 25744 raft_consensus.cc:1050] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:51.848667 25031 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.865151 25744 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.873240 24953 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.873538 24854 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.879063 24948 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.880563 25623 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.867735 25104 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:51.907863 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.912791 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a2ec72789da4a7383ca5ce0f90a4563 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:51.917819 25811 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.921047 25623 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.922435 25801 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:51.932397 25099 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.933410 25800 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.938414 25622 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.939469 25028 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:51.939903 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2ec72789da4a7383ca5ce0f90a4563 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:51.939554 25743 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.941525 25790 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.942823 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.944080 25790 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.949266 25815 tablet_replica.cc:330] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:51.950541 25815 raft_consensus.cc:2229] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:51.952419 25815 raft_consensus.cc:2258] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:51.953454 25785 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:51.954130 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:51.958492 25815 ts_tablet_manager.cc:1875] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:51.960337 25790 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.967072 25031 raft_consensus.cc:2932] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.971464 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4680c139c46f44b49096bfd65f033a32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:51.973887 24851 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.968163 25623 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.985059 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 4680c139c46f44b49096bfd65f033a32 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:51.990953 25743 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:51.996795 25743 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:51.998296 25620 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.005308 24851 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.020814 25099 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.013475 25815 ts_tablet_manager.cc:1888] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.024956 25815 log.cc:1196] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/8a2ec72789da4a7383ca5ce0f90a4563
I20240426 13:18:52.026788 25815 tablet_replica.cc:330] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:52.027670 25815 raft_consensus.cc:2229] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.028321 25815 raft_consensus.cc:2258] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.029111 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 8a2ec72789da4a7383ca5ce0f90a4563 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.030129 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:52.032456 25023 raft_consensus.cc:2932] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.033852 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 786abf3f43724ad5b959e497fc776d1d with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:52.050668 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 786abf3f43724ad5b959e497fc776d1d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:52.063310 25815 ts_tablet_manager.cc:1875] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.066633 25820 tablet_replica.cc:330] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:52.068290 25820 raft_consensus.cc:2229] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.069011 25820 raft_consensus.cc:2258] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.077778 25820 ts_tablet_manager.cc:1875] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.081666 25815 ts_tablet_manager.cc:1888] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.082134 25815 log.cc:1196] T 4680c139c46f44b49096bfd65f033a32 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/4680c139c46f44b49096bfd65f033a32
I20240426 13:18:52.084797 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 4680c139c46f44b49096bfd65f033a32 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.084903 25028 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.097409 25820 ts_tablet_manager.cc:1888] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.097923 25820 log.cc:1196] T 786abf3f43724ad5b959e497fc776d1d P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/786abf3f43724ad5b959e497fc776d1d
I20240426 13:18:52.100286 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 786abf3f43724ad5b959e497fc776d1d (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:52.135727 24951 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:52.141619 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:52.142179 25028 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:52.142972 25818 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:52.143923 25810 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:52.149430 25818 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.153329 25620 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.151984 25028 raft_consensus.cc:2932] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.158563 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3a669ffe77a4ad29604f4ecd7c2644a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:52.168256 24852 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:52.176790 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a669ffe77a4ad29604f4ecd7c2644a with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:52.177583 25820 tablet_replica.cc:330] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:52.178306 25820 raft_consensus.cc:2229] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:52.178740 25820 raft_consensus.cc:2258] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:52.181722 25820 ts_tablet_manager.cc:1875] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:52.193675 25820 ts_tablet_manager.cc:1888] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:52.194017 25820 log.cc:1196] T d3a669ffe77a4ad29604f4ecd7c2644a P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/d3a669ffe77a4ad29604f4ecd7c2644a
I20240426 13:18:52.195362 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet d3a669ffe77a4ad29604f4ecd7c2644a (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:53.225579 24953 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:53.231776 24920 consensus_peers.cc:485] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f49912f064d045d8a1d8be2c08b97d44. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.232656 25028 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.232656 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.234134 25818 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.235033 25810 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.242494 25810 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.244237 25104 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.260104 24951 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.262152 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:53.275815 25104 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:53.276543 24920 consensus_peers.cc:485] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2212db2dbcf486888aefb7dd0174287. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.278766 25824 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.280958 25024 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.283305 25824 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:53.296258 25028 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.298621 25818 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.314462 25104 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.314219 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
W20240426 13:18:53.322115 25069 consensus_peers.cc:485] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3014c46944bc4cbb82624ac909221634. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.325395 25099 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.326581 24951 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.327839 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.330459 25028 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.333269 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.345095 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:53.363379 25801 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.365135 24951 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.374058 25028 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.378552 25099 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.389781 24853 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:18:53.399066 25069 consensus_peers.cc:485] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4010c37c00d94e78b5e3260ad78e7904. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:53.401839 24951 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.402364 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:53.403501 25801 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.405297 25800 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:53.413429 25801 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.414808 24951 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.433481 24851 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.445396 25028 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:53.727800 25842 ts_tablet_manager.cc:902] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:53.731966 25842 tablet_copy_client.cc:308] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:53.734016 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet a2212db2dbcf486888aefb7dd0174287 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:53.734459 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet a2212db2dbcf486888aefb7dd0174287 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287
I20240426 13:18:53.740015 24966 tablet_copy_source_session.cc:205] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.743647 25842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2212db2dbcf486888aefb7dd0174287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.776082 25842 tablet_copy_client.cc:783] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.776814 25842 tablet_copy_client.cc:647] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.780441 25842 tablet_copy_client.cc:523] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.787982 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.811697 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:53.812543 25842 tablet_bootstrap.cc:492] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:53.813133 25842 ts_tablet_manager.cc:1367] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.009s
I20240426 13:18:53.815642 25842 raft_consensus.cc:348] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.816593 25842 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:53.817205 25842 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.820102 25842 ts_tablet_manager.cc:1398] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240426 13:18:53.822152 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:53.822556 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-a2212db2dbcf486888aefb7dd0174287 on tablet a2212db2dbcf486888aefb7dd0174287 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:53.838300 25842 ts_tablet_manager.cc:902] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:53.844012 25842 tablet_copy_client.cc:308] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:53.845317 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet f49912f064d045d8a1d8be2c08b97d44 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:53.845752 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet f49912f064d045d8a1d8be2c08b97d44 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44
I20240426 13:18:53.853088 24966 tablet_copy_source_session.cc:205] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.855602 25842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49912f064d045d8a1d8be2c08b97d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.859638 25853 ts_tablet_manager.cc:902] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:53.867772 25842 tablet_copy_client.cc:783] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.868428 25842 tablet_copy_client.cc:647] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.874514 25842 tablet_copy_client.cc:523] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.882189 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.905387 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:53.906479 25842 tablet_bootstrap.cc:492] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:53.907130 25842 ts_tablet_manager.cc:1367] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20240426 13:18:53.907829 25853 tablet_copy_client.cc:308] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:53.909432 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 4010c37c00d94e78b5e3260ad78e7904 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:53.909919 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 4010c37c00d94e78b5e3260ad78e7904 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904
I20240426 13:18:53.910305 25842 raft_consensus.cc:348] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.911141 25842 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:53.911744 25842 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:53.916289 25115 tablet_copy_source_session.cc:205] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.920512 25842 ts_tablet_manager.cc:1398] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.013s user 0.002s sys 0.004s
I20240426 13:18:53.923949 25859 ts_tablet_manager.cc:902] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:53.930099 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:53.930716 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-f49912f064d045d8a1d8be2c08b97d44 on tablet f49912f064d045d8a1d8be2c08b97d44 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:53.935576 25853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4010c37c00d94e78b5e3260ad78e7904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.944118 25859 tablet_copy_client.cc:308] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:53.945808 25115 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 3014c46944bc4cbb82624ac909221634 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:53.946328 25115 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 3014c46944bc4cbb82624ac909221634 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634
I20240426 13:18:53.954093 25115 tablet_copy_source_session.cc:205] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:53.954476 25853 tablet_copy_client.cc:783] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.955070 25853 tablet_copy_client.cc:647] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.958595 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3014c46944bc4cbb82624ac909221634. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:53.959684 25853 tablet_copy_client.cc:523] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:53.973110 25859 tablet_copy_client.cc:783] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:53.973718 25859 tablet_copy_client.cc:647] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:53.976321 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:53.992342 25859 tablet_copy_client.cc:523] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:54.013170 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:54.014508 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:54.015527 25853 tablet_bootstrap.cc:492] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:54.016276 25853 ts_tablet_manager.cc:1367] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.021s
I20240426 13:18:54.019222 25853 raft_consensus.cc:348] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.020246 25853 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:54.020962 25853 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.028759 25853 ts_tablet_manager.cc:1398] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20240426 13:18:54.031286 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:54.031826 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-4010c37c00d94e78b5e3260ad78e7904 on tablet 4010c37c00d94e78b5e3260ad78e7904 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:54.045656 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:54.046636 25859 tablet_bootstrap.cc:492] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:54.047314 25859 ts_tablet_manager.cc:1367] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.008s
I20240426 13:18:54.049947 25859 raft_consensus.cc:348] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.050743 25859 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:54.051349 25859 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:54.053966 25859 ts_tablet_manager.cc:1398] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240426 13:18:54.055462 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:54.055840 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-3014c46944bc4cbb82624ac909221634 on tablet 3014c46944bc4cbb82624ac909221634 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:54.143039 25623 raft_consensus.cc:1203] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:54.223541 25623 raft_consensus.cc:1203] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:54.388173 25623 raft_consensus.cc:1203] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:54.447331 25623 raft_consensus.cc:1203] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:54.626962 25866 raft_consensus.cc:1050] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.629279 25866 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.637151 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.637681 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.638707 25824 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.639130 25024 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.640287 25866 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.641127 25824 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.649780 25831 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.652242 25623 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.663743 25024 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.667017 24852 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.689015 25099 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.710702 25866 raft_consensus.cc:1050] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.714866 25866 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.721395 25623 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.722862 25104 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.724560 25866 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.726260 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.726392 25831 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.730122 25831 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.739257 24951 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.745576 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.746022 25099 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.746927 25871 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.747920 25099 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.751405 25872 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.742640 25831 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.753758 25873 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.747700 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.756342 25623 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.756563 25105 raft_consensus.cc:2932] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.768644 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f49912f064d045d8a1d8be2c08b97d44 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:54.771919 25623 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.775703 24851 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.777936 24853 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.806648 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet f49912f064d045d8a1d8be2c08b97d44 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:54.813918 25883 tablet_replica.cc:330] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:54.813331 24951 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.815016 25883 raft_consensus.cc:2229] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:54.820977 25883 raft_consensus.cc:2258] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:54.824467 25883 ts_tablet_manager.cc:1875] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:54.830653 25623 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.833101 25866 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.834162 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.835690 25872 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.836591 25800 raft_consensus.cc:1050] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:54.838806 25800 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.843976 25866 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.848352 25623 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.854110 25028 raft_consensus.cc:2932] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.857388 25883 ts_tablet_manager.cc:1888] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:54.857810 25883 log.cc:1196] T f49912f064d045d8a1d8be2c08b97d44 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/f49912f064d045d8a1d8be2c08b97d44
I20240426 13:18:54.859179 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet f49912f064d045d8a1d8be2c08b97d44 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:54.861816 25620 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.863267 25024 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.863643 24954 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:54.865172 25801 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.866063 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.868155 25800 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:54.871729 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2212db2dbcf486888aefb7dd0174287 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:54.878091 24852 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.892760 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet a2212db2dbcf486888aefb7dd0174287 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:54.896920 25887 tablet_replica.cc:330] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:54.902024 25887 raft_consensus.cc:2229] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:54.902549 25887 raft_consensus.cc:2258] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:54.906528 25887 ts_tablet_manager.cc:1875] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:54.918270 25800 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.925550 25623 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.928622 25887 ts_tablet_manager.cc:1888] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:54.929145 25887 log.cc:1196] T a2212db2dbcf486888aefb7dd0174287 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/a2212db2dbcf486888aefb7dd0174287
I20240426 13:18:54.930431 25024 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.930846 24954 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.937450 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet a2212db2dbcf486888aefb7dd0174287 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:54.946561 24852 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:18:54.963316 25099 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:54.969354 24954 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.972153 25623 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:54.971413 25800 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:54.973387 25825 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:54.980775 25801 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.983352 25623 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:54.987736 24954 raft_consensus.cc:2932] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.003703 25825 raft_consensus.cc:1050] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:55.003402 24853 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.006012 25825 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:55.008661 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3014c46944bc4cbb82624ac909221634 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:55.018595 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3014c46944bc4cbb82624ac909221634 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:55.019615 25883 tablet_replica.cc:330] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:55.022804 25623 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.023370 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.024838 25885 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.027278 25885 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.032789 24953 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:55.034003 25825 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:55.047252 25890 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.052565 25028 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.057929 25883 raft_consensus.cc:2229] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:55.058701 25883 raft_consensus.cc:2258] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:55.059232 25622 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.061921 25883 ts_tablet_manager.cc:1875] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:55.067788 24954 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.074064 24853 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.095669 25883 ts_tablet_manager.cc:1888] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:55.096220 25883 log.cc:1196] T 3014c46944bc4cbb82624ac909221634 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/3014c46944bc4cbb82624ac909221634
I20240426 13:18:55.097862 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 3014c46944bc4cbb82624ac909221634 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:55.107892 25105 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:55.112452 25028 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:55.114296 25622 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:55.114290 25890 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:55.117297 25890 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:18:55.121201 25801 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.122189 25024 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.127765 25622 raft_consensus.cc:2932] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.130074 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4010c37c00d94e78b5e3260ad78e7904 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:55.140839 24853 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:55.154361 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 4010c37c00d94e78b5e3260ad78e7904 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:55.164561 25895 tablet_replica.cc:330] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:55.167372 25895 raft_consensus.cc:2229] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:55.168115 25895 raft_consensus.cc:2258] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:55.170822 25895 ts_tablet_manager.cc:1875] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:55.182121 25895 ts_tablet_manager.cc:1888] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:55.182444 25895 log.cc:1196] T 4010c37c00d94e78b5e3260ad78e7904 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/4010c37c00d94e78b5e3260ad78e7904
I20240426 13:18:55.183744 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 4010c37c00d94e78b5e3260ad78e7904 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:56.196419 25024 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.202286 25105 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:56.202785 24995 consensus_peers.cc:485] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59632e528afa44ca98a626355e1e4660. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.203804 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.204438 24954 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.206842 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.213286 25841 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.217233 25105 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.219576 24953 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.230674 24852 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.234701 25024 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:56.242067 24995 consensus_peers.cc:485] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fbdcc6d88d14934b64bc2a29382519f. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.242452 24954 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.244304 25105 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.244347 25751 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.245747 25900 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.256537 25841 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.257953 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.259173 24954 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.276935 25099 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.284691 25024 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.285636 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.287264 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:56.288563 25069 consensus_peers.cc:485] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f413a75d3c249dbb47c1ec61978c2ea. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.287873 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.289350 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.306808 25825 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.309729 24954 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.313838 25028 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.326049 24953 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:56.333786 24920 consensus_peers.cc:485] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be16e3d2b66a44c5ac4a072b891540a2. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:56.335512 25099 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.335413 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.336259 25028 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:56.336956 25824 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.337874 25866 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:56.355154 25866 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.359668 25099 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.364403 25028 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:56.373754 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:56.624153 25916 ts_tablet_manager.cc:902] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:56.630928 25916 tablet_copy_client.cc:308] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:56.632521 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 59632e528afa44ca98a626355e1e4660 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:56.633020 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 59632e528afa44ca98a626355e1e4660 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660
I20240426 13:18:56.639328 25041 tablet_copy_source_session.cc:205] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.644604 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59632e528afa44ca98a626355e1e4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.658077 25916 tablet_copy_client.cc:783] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.658564 25916 tablet_copy_client.cc:647] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.661454 25916 tablet_copy_client.cc:523] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.668095 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.692282 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.693157 25916 tablet_bootstrap.cc:492] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.693761 25916 ts_tablet_manager.cc:1367] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.000s
I20240426 13:18:56.696115 25916 raft_consensus.cc:348] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.696686 25916 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.697307 25916 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.699898 25916 ts_tablet_manager.cc:1398] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240426 13:18:56.701231 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:56.701572 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-59632e528afa44ca98a626355e1e4660 on tablet 59632e528afa44ca98a626355e1e4660 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.820650 25916 ts_tablet_manager.cc:902] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:56.824151 25916 tablet_copy_client.cc:308] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:56.825520 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 0fbdcc6d88d14934b64bc2a29382519f from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:56.826016 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 0fbdcc6d88d14934b64bc2a29382519f from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f
I20240426 13:18:56.832777 25041 tablet_copy_source_session.cc:205] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.835893 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbdcc6d88d14934b64bc2a29382519f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.849298 25916 tablet_copy_client.cc:783] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.849869 25916 tablet_copy_client.cc:647] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.853013 25916 tablet_copy_client.cc:523] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.860098 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.869571 25927 ts_tablet_manager.cc:902] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:56.886387 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.887337 25916 tablet_bootstrap.cc:492] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.888073 25916 ts_tablet_manager.cc:1367] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.001s
I20240426 13:18:56.890434 25916 raft_consensus.cc:348] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.891256 25916 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.891929 25916 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.894686 25927 tablet_copy_client.cc:308] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:56.894724 25916 ts_tablet_manager.cc:1398] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240426 13:18:56.899391 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 7f413a75d3c249dbb47c1ec61978c2ea from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:56.899883 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 7f413a75d3c249dbb47c1ec61978c2ea from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea
I20240426 13:18:56.900571 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:56.901022 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-0fbdcc6d88d14934b64bc2a29382519f on tablet 0fbdcc6d88d14934b64bc2a29382519f with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.906963 25114 tablet_copy_source_session.cc:205] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.910073 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f413a75d3c249dbb47c1ec61978c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.913739 25916 ts_tablet_manager.cc:902] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:18:56.922430 25927 tablet_copy_client.cc:783] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.922914 25927 tablet_copy_client.cc:647] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.926467 25927 tablet_copy_client.cc:523] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.928491 25916 tablet_copy_client.cc:308] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:18:56.931247 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet be16e3d2b66a44c5ac4a072b891540a2 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:18:56.931849 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet be16e3d2b66a44c5ac4a072b891540a2 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2
I20240426 13:18:56.940616 24966 tablet_copy_source_session.cc:205] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:56.948838 25916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be16e3d2b66a44c5ac4a072b891540a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:56.959020 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.962914 25916 tablet_copy_client.cc:783] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:56.963443 25916 tablet_copy_client.cc:647] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:56.968056 25916 tablet_copy_client.cc:523] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:56.975446 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:56.977679 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:56.978585 25927 tablet_bootstrap.cc:492] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:56.979209 25927 ts_tablet_manager.cc:1367] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240426 13:18:56.981926 25927 raft_consensus.cc:348] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.982748 25927 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:56.983341 25927 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:56.986285 25927 ts_tablet_manager.cc:1398] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240426 13:18:56.987740 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea received from {username='slave'} at 127.0.0.1:55910
I20240426 13:18:56.988152 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-7f413a75d3c249dbb47c1ec61978c2ea on tablet 7f413a75d3c249dbb47c1ec61978c2ea with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:56.999377 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:57.000203 25916 tablet_bootstrap.cc:492] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:57.001224 25916 ts_tablet_manager.cc:1367] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.001s
I20240426 13:18:57.003489 25916 raft_consensus.cc:348] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:57.004298 25916 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:57.004819 25916 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:57.006474 25916 ts_tablet_manager.cc:1398] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240426 13:18:57.007714 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2 received from {username='slave'} at 127.0.0.1:46260
I20240426 13:18:57.008260 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-be16e3d2b66a44c5ac4a072b891540a2 on tablet be16e3d2b66a44c5ac4a072b891540a2 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:57.151690 25622 raft_consensus.cc:1203] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:57.229954 25622 raft_consensus.cc:1203] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:57.342352 25622 raft_consensus.cc:1203] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:57.484723 25622 raft_consensus.cc:1203] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:18:57.615234 25939 raft_consensus.cc:1050] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.617318 25939 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.622464 25622 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.622838 25099 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.623359 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.623736 25751 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.624914 25940 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.625731 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.644732 25751 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.646504 25099 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.656915 25622 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.657617 24954 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.668888 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.700315 25031 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.706569 24953 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.707881 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.717337 25940 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.718456 24954 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.721096 25622 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.722338 25939 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.727564 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59632e528afa44ca98a626355e1e4660 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:57.729221 25622 raft_consensus.cc:2932] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.737557 24851 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:57.753280 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 59632e528afa44ca98a626355e1e4660 with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:18:57.767735 25951 tablet_replica.cc:330] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:18:57.768839 25951 raft_consensus.cc:2229] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:57.769374 25951 raft_consensus.cc:2258] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:57.771883 25951 ts_tablet_manager.cc:1875] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:57.793289 25951 ts_tablet_manager.cc:1888] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:57.793687 25951 log.cc:1196] T 59632e528afa44ca98a626355e1e4660 P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/59632e528afa44ca98a626355e1e4660
I20240426 13:18:57.794857 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 59632e528afa44ca98a626355e1e4660 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:57.838855 25940 raft_consensus.cc:1050] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.840932 25940 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.846885 25622 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.847671 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.849395 24954 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.849951 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:18:57.850947 25940 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.863031 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.867779 25939 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.868891 25622 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.876675 24953 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.878646 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.881634 24854 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.898689 25959 raft_consensus.cc:1050] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:57.910522 25024 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.908663 25959 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:57.915374 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.916553 25955 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.919689 25622 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:57.925176 25939 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:57.925611 25105 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.925060 25940 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.923885 25622 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.926489 25099 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.927536 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.929574 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.935375 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fbdcc6d88d14934b64bc2a29382519f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:57.936209 25622 raft_consensus.cc:2932] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.941623 25031 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:57.944528 25959 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:57.947400 24852 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:57.960638 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbdcc6d88d14934b64bc2a29382519f with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:18:57.987248 25824 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:57.989207 25099 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.005076 25622 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.006508 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.020666 25964 tablet_replica.cc:330] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:18:58.020277 25028 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.022183 25964 raft_consensus.cc:2229] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.022907 25964 raft_consensus.cc:2258] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.026538 25964 ts_tablet_manager.cc:1875] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.064466 24954 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.073145 25622 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.074669 25824 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.079007 25964 ts_tablet_manager.cc:1888] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.079517 25964 log.cc:1196] T 0fbdcc6d88d14934b64bc2a29382519f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/0fbdcc6d88d14934b64bc2a29382519f
I20240426 13:18:58.081748 25622 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.080405 25959 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.083606 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.086620 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 0fbdcc6d88d14934b64bc2a29382519f (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:58.087888 25942 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.094753 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be16e3d2b66a44c5ac4a072b891540a2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:58.095463 25801 raft_consensus.cc:1050] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:18:58.097676 25801 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.098747 25104 raft_consensus.cc:2932] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.109737 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.109627 24852 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.111053 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.111142 25028 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.112442 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.116168 25622 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:18:58.117586 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:18:58.128346 25825 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.130404 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet be16e3d2b66a44c5ac4a072b891540a2 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:58.132359 25969 tablet_replica.cc:330] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:58.133339 25969 raft_consensus.cc:2229] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.134076 25969 raft_consensus.cc:2258] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.135082 25028 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.138366 25969 ts_tablet_manager.cc:1875] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.142772 25622 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.159469 25969 ts_tablet_manager.cc:1888] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.159924 25969 log.cc:1196] T be16e3d2b66a44c5ac4a072b891540a2 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/be16e3d2b66a44c5ac4a072b891540a2
I20240426 13:18:58.160482 24953 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.165184 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet be16e3d2b66a44c5ac4a072b891540a2 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:58.169804 24851 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.189563 25105 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:18:58.194751 25622 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.194986 24954 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:18:58.195974 25825 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.196753 25801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:18:58.201848 25801 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.206001 25622 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.209954 24954 raft_consensus.cc:2932] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.211107 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f413a75d3c249dbb47c1ec61978c2ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:18:58.232669 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:18:58.245393 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f413a75d3c249dbb47c1ec61978c2ea with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:18:58.246258 25969 tablet_replica.cc:330] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:18:58.247165 25969 raft_consensus.cc:2229] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:18:58.247748 25969 raft_consensus.cc:2258] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:18:58.250881 25969 ts_tablet_manager.cc:1875] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:18:58.265786 25969 ts_tablet_manager.cc:1888] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:18:58.266142 25969 log.cc:1196] T 7f413a75d3c249dbb47c1ec61978c2ea P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/7f413a75d3c249dbb47c1ec61978c2ea
I20240426 13:18:58.267271 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 7f413a75d3c249dbb47c1ec61978c2ea (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:18:59.301422 25024 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:59.309525 24995 consensus_peers.cc:485] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d066b73b2e464d80b7b667e05602d879. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.309827 24954 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.310523 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.311933 25971 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.313211 25940 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.322439 24951 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.321236 25971 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.326931 25104 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.344908 24852 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.345774 25028 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:18:59.362449 24995 consensus_peers.cc:485] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edc419d5057b449c8d0888a41017ce2e. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.362493 24951 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.364362 25978 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.366495 25104 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.375761 25751 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.376580 24951 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.376644 25971 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.386862 25099 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.396835 25104 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.397182 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.411660 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.413062 25825 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:18:59.418092 25069 consensus_peers.cc:485] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99a876ca7b6e4082b58359403a47730d. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.420697 24951 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.422183 25801 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.437270 25825 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.438686 25028 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.440394 24951 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.455523 25104 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.456389 24853 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:18:59.468987 25028 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:18:59.470305 25069 consensus_peers.cc:485] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0238fe3803ea4654ae30109aa503e3a3. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:18:59.470672 25970 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:18:59.471460 24951 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:18:59.473116 25970 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:18:59.481915 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.490777 24954 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.491387 25028 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.508062 24854 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:18:59.770419 25993 ts_tablet_manager.cc:902] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:18:59.775177 25993 tablet_copy_client.cc:308] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:18:59.776942 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet d066b73b2e464d80b7b667e05602d879 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:18:59.777524 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet d066b73b2e464d80b7b667e05602d879 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879
I20240426 13:18:59.784606 25041 tablet_copy_source_session.cc:205] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:59.787808 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d066b73b2e464d80b7b667e05602d879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:59.802915 25993 tablet_copy_client.cc:783] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:59.803588 25993 tablet_copy_client.cc:647] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:59.808915 25993 tablet_copy_client.cc:523] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:59.819212 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:59.840114 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:18:59.841004 25993 tablet_bootstrap.cc:492] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:18:59.841676 25993 ts_tablet_manager.cc:1367] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.005s
I20240426 13:18:59.844214 25993 raft_consensus.cc:348] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.845171 25993 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:18:59.845824 25993 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:18:59.849867 25993 ts_tablet_manager.cc:1398] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20240426 13:18:59.851168 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:18:59.851574 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-d066b73b2e464d80b7b667e05602d879 on tablet d066b73b2e464d80b7b667e05602d879 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:18:59.911662 25993 ts_tablet_manager.cc:902] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:59.925277 25993 tablet_copy_client.cc:308] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:18:59.927604 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 0238fe3803ea4654ae30109aa503e3a3 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:18:59.928133 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 0238fe3803ea4654ae30109aa503e3a3 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3
I20240426 13:18:59.936527 25114 tablet_copy_source_session.cc:205] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:18:59.939178 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0238fe3803ea4654ae30109aa503e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:18:59.955466 25993 tablet_copy_client.cc:783] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:18:59.957197 25993 tablet_copy_client.cc:647] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:18:59.957490 26004 ts_tablet_manager.cc:902] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:18:59.962553 25993 tablet_copy_client.cc:523] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:18:59.971526 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:18:59.977526 26007 ts_tablet_manager.cc:902] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:00.002734 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.003829 25993 tablet_bootstrap.cc:492] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.004698 25993 ts_tablet_manager.cc:1367] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.009s
I20240426 13:19:00.012919 26004 tablet_copy_client.cc:308] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:00.074890 25993 raft_consensus.cc:348] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.075839 25993 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.076503 25993 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.083178 25993 ts_tablet_manager.cc:1398] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.078s user 0.040s sys 0.034s
I20240426 13:19:00.101538 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 99a876ca7b6e4082b58359403a47730d from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:00.102150 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 99a876ca7b6e4082b58359403a47730d from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d
I20240426 13:19:00.109226 25115 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:00.109639 25115 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-0238fe3803ea4654ae30109aa503e3a3 on tablet 0238fe3803ea4654ae30109aa503e3a3 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.114236 26007 tablet_copy_client.cc:308] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:00.116364 25114 tablet_copy_source_session.cc:205] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:00.118003 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet edc419d5057b449c8d0888a41017ce2e from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:00.118533 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet edc419d5057b449c8d0888a41017ce2e from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e
I20240426 13:19:00.120348 26004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a876ca7b6e4082b58359403a47730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:00.133128 25041 tablet_copy_source_session.cc:205] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:00.140252 26007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc419d5057b449c8d0888a41017ce2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:00.143402 26004 tablet_copy_client.cc:783] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:00.143981 26004 tablet_copy_client.cc:647] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:00.148052 26004 tablet_copy_client.cc:523] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:00.153106 26007 tablet_copy_client.cc:783] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:00.153693 26007 tablet_copy_client.cc:647] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:00.156414 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:00.157876 26007 tablet_copy_client.cc:523] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:00.171816 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:00.177421 25623 raft_consensus.cc:1203] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:00.214015 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.215214 26004 tablet_bootstrap.cc:492] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.216128 26004 ts_tablet_manager.cc:1367] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.060s user 0.032s sys 0.013s
I20240426 13:19:00.220667 26004 raft_consensus.cc:348] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.221599 26004 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.222246 26004 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.227188 26004 ts_tablet_manager.cc:1398] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20240426 13:19:00.227279 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:00.228178 26007 tablet_bootstrap.cc:492] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:00.228725 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:00.228893 26007 ts_tablet_manager.cc:1367] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.019s
I20240426 13:19:00.229126 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-99a876ca7b6e4082b58359403a47730d on tablet 99a876ca7b6e4082b58359403a47730d with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.231523 26007 raft_consensus.cc:348] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.232365 26007 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:00.232934 26007 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:00.235967 26007 ts_tablet_manager.cc:1398] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240426 13:19:00.241431 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:00.241825 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-edc419d5057b449c8d0888a41017ce2e on tablet edc419d5057b449c8d0888a41017ce2e with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:00.333727 25623 raft_consensus.cc:1203] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:00.394454 25623 raft_consensus.cc:1203] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:00.508931 25623 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:00.686053 26016 raft_consensus.cc:1050] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.688557 26016 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.695690 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.696221 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.696442 24951 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.698252 25751 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.698931 25971 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.699810 26016 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.723372 26016 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.724686 25623 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.732304 25099 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.734202 24951 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.737568 24854 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.767899 25028 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.773276 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.774092 25104 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.776093 25751 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.776795 26016 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.785106 26016 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.795241 25620 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.796653 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d066b73b2e464d80b7b667e05602d879 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:00.801748 25986 raft_consensus.cc:1050] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.804095 25986 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.813707 25623 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.813014 24853 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:00.815106 25100 raft_consensus.cc:2932] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.816970 25028 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.818151 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.827376 25825 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:00.834836 24954 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:00.836184 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:00.840636 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet d066b73b2e464d80b7b667e05602d879 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:00.848726 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.851570 26033 tablet_replica.cc:330] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:00.852692 26033 raft_consensus.cc:2229] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:00.854810 25031 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.864002 26033 raft_consensus.cc:2258] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:00.865681 25620 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.867722 24954 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.875187 26033 ts_tablet_manager.cc:1875] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:00.879092 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.889506 26033 ts_tablet_manager.cc:1888] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:00.889925 26033 log.cc:1196] T d066b73b2e464d80b7b667e05602d879 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/d066b73b2e464d80b7b667e05602d879
I20240426 13:19:00.892160 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet d066b73b2e464d80b7b667e05602d879 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:00.912358 25100 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:00.919200 25623 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.920692 26018 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.921202 24951 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:00.922423 25986 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:00.926403 25825 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.928972 25623 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.935842 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0238fe3803ea4654ae30109aa503e3a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:00.937171 24951 raft_consensus.cc:2932] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.939442 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:00.950425 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0238fe3803ea4654ae30109aa503e3a3 with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:00.960610 26035 tablet_replica.cc:330] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:00.961799 26035 raft_consensus.cc:2229] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:00.962443 26035 raft_consensus.cc:2258] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:00.969537 26035 ts_tablet_manager.cc:1875] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:00.981601 26035 ts_tablet_manager.cc:1888] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:00.981959 26035 log.cc:1196] T 0238fe3803ea4654ae30109aa503e3a3 P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/0238fe3803ea4654ae30109aa503e3a3
I20240426 13:19:00.983361 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet 0238fe3803ea4654ae30109aa503e3a3 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:00.995960 26016 raft_consensus.cc:1050] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:00.998164 26016 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.003784 25100 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.004099 24951 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.004230 25623 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.005329 25971 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.006354 26016 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.007784 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.027873 25971 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.029115 25099 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.036195 25623 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.042851 24852 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:01.046979 24951 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.057014 25028 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.069000 25623 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.070199 24954 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.070216 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:01.073226 26029 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.078960 26016 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.082155 25620 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.093313 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edc419d5057b449c8d0888a41017ce2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:01.098066 24954 raft_consensus.cc:2932] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.102898 24851 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:01.117213 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet edc419d5057b449c8d0888a41017ce2e with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:01.122251 26041 tablet_replica.cc:330] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:01.134954 25825 raft_consensus.cc:1050] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:01.136847 25825 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.141232 26041 raft_consensus.cc:2229] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:01.141939 26041 raft_consensus.cc:2258] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:01.142513 25620 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.144532 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:01.151207 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.152546 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.153234 26041 ts_tablet_manager.cc:1875] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:01.160465 24954 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:01.163919 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:01.176910 25825 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.178527 24954 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.184266 25620 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.186139 26041 ts_tablet_manager.cc:1888] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:01.186646 26041 log.cc:1196] T edc419d5057b449c8d0888a41017ce2e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/edc419d5057b449c8d0888a41017ce2e
I20240426 13:19:01.188800 25028 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.198423 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.207475 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet edc419d5057b449c8d0888a41017ce2e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:01.219924 25099 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:01.224910 25620 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.225299 25023 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:01.226051 25986 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.229921 26046 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:01.236726 26046 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.239255 25620 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.242074 25023 raft_consensus.cc:2932] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.245644 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99a876ca7b6e4082b58359403a47730d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:01.251893 24851 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:01.261547 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a876ca7b6e4082b58359403a47730d with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:01.262354 26033 tablet_replica.cc:330] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:01.263240 26033 raft_consensus.cc:2229] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:01.263768 26033 raft_consensus.cc:2258] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:01.266338 26033 ts_tablet_manager.cc:1875] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:01.280411 26033 ts_tablet_manager.cc:1888] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:01.280718 26033 log.cc:1196] T 99a876ca7b6e4082b58359403a47730d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/99a876ca7b6e4082b58359403a47730d
I20240426 13:19:01.281839 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 99a876ca7b6e4082b58359403a47730d (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:02.309391 24954 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.315255 25028 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.315665 24920 consensus_peers.cc:485] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 256415e42b6c444387115ab766a5a16c. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.316437 26048 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.316906 25099 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.318037 25824 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.323326 26048 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.324843 25028 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.325428 25099 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.339781 25099 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.341650 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.347852 24954 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.348134 25069 consensus_peers.cc:485] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c896b58b7766401eac0a7f69574827f1. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.349961 25825 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.355594 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.357256 26018 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.377382 26053 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.378473 25028 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.384641 24954 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.402824 25028 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.419397 24852 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:02.421048 25099 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.421459 24995 consensus_peers.cc:485] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6277899576264e68aa43514e5b5c676e. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.422498 26029 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.425060 24954 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.426335 26029 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.439483 26062 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.447389 25099 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.457667 25100 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.459669 24853 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:02.485332 25028 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:02.486357 25069 consensus_peers.cc:485] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9910a5336aa4e3cab11dca4a556fabf. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:02.486891 26018 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.487866 24954 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.491672 24951 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:02.493253 26053 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:02.501196 26052 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.502843 25028 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.503041 24951 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.519569 24853 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:02.800745 26067 ts_tablet_manager.cc:902] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:02.804800 26067 tablet_copy_client.cc:308] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:02.806668 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 6277899576264e68aa43514e5b5c676e from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:02.807210 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 6277899576264e68aa43514e5b5c676e from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e
I20240426 13:19:02.812971 25041 tablet_copy_source_session.cc:205] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.815369 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6277899576264e68aa43514e5b5c676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.828485 26067 tablet_copy_client.cc:783] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.829033 26067 tablet_copy_client.cc:647] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.833396 26067 tablet_copy_client.cc:523] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.840179 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:02.857419 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:02.858219 26067 tablet_bootstrap.cc:492] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:02.858826 26067 ts_tablet_manager.cc:1367] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20240426 13:19:02.861249 26067 raft_consensus.cc:348] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.861819 26067 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:02.862378 26067 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.865396 26067 ts_tablet_manager.cc:1398] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20240426 13:19:02.867390 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:02.867842 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-6277899576264e68aa43514e5b5c676e on tablet 6277899576264e68aa43514e5b5c676e with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:02.869371 26074 ts_tablet_manager.cc:902] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:02.875453 26074 tablet_copy_client.cc:308] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:02.876981 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet c896b58b7766401eac0a7f69574827f1 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:02.877388 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet c896b58b7766401eac0a7f69574827f1 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1
I20240426 13:19:02.883700 25114 tablet_copy_source_session.cc:205] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.886911 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896b58b7766401eac0a7f69574827f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.898563 26074 tablet_copy_client.cc:783] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.899091 26074 tablet_copy_client.cc:647] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.902982 26074 tablet_copy_client.cc:523] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.906081 26067 ts_tablet_manager.cc:902] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:02.912423 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:02.936269 26067 tablet_copy_client.cc:308] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:02.938190 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 256415e42b6c444387115ab766a5a16c from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:02.938753 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 256415e42b6c444387115ab766a5a16c from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c
I20240426 13:19:02.948165 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:02.949070 26074 tablet_bootstrap.cc:492] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:02.949703 26074 ts_tablet_manager.cc:1367] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.014s
I20240426 13:19:02.950698 24966 tablet_copy_source_session.cc:205] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:02.953358 26074 raft_consensus.cc:348] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.954231 26074 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:02.954515 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256415e42b6c444387115ab766a5a16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:02.954881 26074 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:02.957536 26074 ts_tablet_manager.cc:1398] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240426 13:19:02.958961 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1 received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:02.959333 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-c896b58b7766401eac0a7f69574827f1 on tablet c896b58b7766401eac0a7f69574827f1 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:02.974996 26067 tablet_copy_client.cc:783] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:02.976105 26067 tablet_copy_client.cc:647] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:02.979594 26067 tablet_copy_client.cc:523] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:02.989573 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:03.008121 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:03.008913 26067 tablet_bootstrap.cc:492] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:03.009466 26067 ts_tablet_manager.cc:1367] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20240426 13:19:03.011767 26067 raft_consensus.cc:348] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.012408 26067 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:03.013116 26067 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.017791 26067 ts_tablet_manager.cc:1398] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240426 13:19:03.019201 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:03.019588 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-256415e42b6c444387115ab766a5a16c on tablet 256415e42b6c444387115ab766a5a16c with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:03.061695 26067 ts_tablet_manager.cc:902] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:03.065271 26067 tablet_copy_client.cc:308] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:03.066452 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet b9910a5336aa4e3cab11dca4a556fabf from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:03.066872 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet b9910a5336aa4e3cab11dca4a556fabf from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf
I20240426 13:19:03.076575 25114 tablet_copy_source_session.cc:205] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:03.079156 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9910a5336aa4e3cab11dca4a556fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:03.090981 26067 tablet_copy_client.cc:783] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:03.091554 26067 tablet_copy_client.cc:647] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:03.100021 26067 tablet_copy_client.cc:523] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:03.108280 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:03.125591 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:03.126271 26067 tablet_bootstrap.cc:492] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:03.126857 26067 ts_tablet_manager.cc:1367] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.001s
I20240426 13:19:03.128942 26067 raft_consensus.cc:348] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.129686 26067 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:03.130108 26067 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:03.131898 26067 ts_tablet_manager.cc:1398] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240426 13:19:03.134406 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:03.134795 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-b9910a5336aa4e3cab11dca4a556fabf on tablet b9910a5336aa4e3cab11dca4a556fabf with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:04.197486 26099 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.198417 26099 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.201028 26099 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.201491 26098 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.202109 26098 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.206020 26098 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.247025 26102 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.247726 26102 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.252483 26102 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.254750 26101 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.255283 26101 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.257158 26101 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.273586 26106 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.274185 26106 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.288235 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.288800 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1080) took 1061 ms. Trace:
I20240426 13:19:04.288947 26103 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.289484 26103 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.290330 26106 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.289299 25023 rpcz_store.cc:276] 0426 13:19:03.227760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:03.227949 (+ 189us) service_pool.cc:225] Handling call
0426 13:19:03.348179 (+120230us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:03.348371 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:03.348399 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:03.348447 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:03.348480 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:04.288543 (+940063us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:04.288708 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:04.289501 26101 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.294337 26096 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.295020 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1081) took 1065 ms. Trace:
I20240426 13:19:04.294939 26096 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.295293 25028 rpcz_store.cc:276] 0426 13:19:03.229935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:03.230055 (+ 120us) service_pool.cc:225] Handling call
0426 13:19:04.294545 (+1064490us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:04.294702 (+ 157us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:04.294815 (+ 113us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:04.294831 (+ 16us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:04.294856 (+ 25us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:04.294875 (+ 19us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:04.294912 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:04.294980 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4982528}
I20240426 13:19:04.291577 26103 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.297641 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:04.304328 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.0225e+06us
I20240426 13:19:04.305758 26106 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.306264 26106 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.308394 26106 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.311071 26106 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.311612 26106 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.314294 26106 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.315784 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.316442 25104 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:04.316767 26100 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.316932 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.317400 26100 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.292235 26101 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.320621 26100 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.322876 26100 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.323536 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.323360 26100 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.324473 26101 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.325408 26100 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.326151 26100 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.326680 26100 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.328702 26100 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.329471 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.296869 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.301607 26098 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.302075 26097 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.331400 26097 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.331462 26098 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.333216 26101 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.333348 26097 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.302997 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.333711 26101 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.299479 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.318684 25621 raft_consensus.cc:1203] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:04.334723 24953 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.318881 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:04.338172 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.297847 26096 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.335785 26101 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.345084 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.345180 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.344383 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.345489 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.342612 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.348901 25620 raft_consensus.cc:1203] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:04.347630 26107 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.347735 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.356138 26096 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.345916 26106 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.362097 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.362134 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.362860 26096 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.364224 26107 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.374132 26096 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.374284 26107 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.389904 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.390844 25099 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.360368 26106 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.470813 26095 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.471333 26095 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.486956 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.465292 26106 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.461514 26107 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.497352 26103 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.496608 25618 raft_consensus.cc:1203] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:04.464810 26102 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.470120 26098 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.469780 26097 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.522184 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:04.485286 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 140220us
I20240426 13:19:04.548730 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.554600 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 201690us
I20240426 13:19:04.519831 26112 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.563189 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:04.525398 26103 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:04.518607 26095 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.521283 26113 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:04.569607 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 205988us
I20240426 13:19:04.523006 26107 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.575383 26114 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.525080 26102 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.571854 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.576015 26104 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.576542 26104 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.578939 26103 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.569164 26098 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:04.567718 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 205688us
I20240426 13:19:04.582087 26104 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.582321 26102 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.562381 26097 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.597486 26115 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.598101 26109 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.570482 26112 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.573710 26101 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.581295 26098 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.600384 26116 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.608480 26119 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.608826 26109 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.612102 26109 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.601253 26104 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.613278 26104 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.615643 26107 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.616248 26107 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.617113 26112 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.616818 26107 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.617903 26117 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.605686 26098 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.643393 26112 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:04.644124 26112 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.646576 26112 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.672863 26112 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.673579 26112 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.618387 26117 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.607510 26115 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.609061 26119 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.619180 26114 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.704139 26118 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.574576 26113 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.605481 26101 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.747735 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.757802 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.711843 26117 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.710330 26115 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.704849 26106 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.762382 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:04.766434 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.740432 26116 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.769995 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.743670 26110 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:04.774116 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75082.6us
I20240426 13:19:04.775053 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.776927 26112 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.777052 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.777601 26101 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.772768 26117 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.798383 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.712738 26114 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.742097 26118 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.777601 26110 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.778503 26104 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.772989 26116 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.782083 26094 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.877566 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.936910 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.904229 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 141961us
I20240426 13:19:04.942623 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.943544 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.834307 26106 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.781936 25615 raft_consensus.cc:1203] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:04.944407 25618 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.946386 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.805853 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.947501 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.947144 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:04.902905 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 304936us
I20240426 13:19:04.874433 26126 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.949040 26126 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.950109 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:04.951393 26127 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.951610 26126 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.875815 26119 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.879195 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.952863 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.953190 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.877506 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:04.960424 26132 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.961084 26132 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.833194 26123 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.961558 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.953505 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.969955 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.970152 25618 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.874239 26128 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.880482 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:04.971071 26128 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.878769 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.973205 26128 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.955242 25620 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:04.955320 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.956434 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.957830 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.975557 25097 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:04.975513 25618 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:04.798013 26097 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:04.959303 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:04.834525 26121 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.981076 26130 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.980950 26121 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.981336 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:04.981536 26130 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.981971 26130 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:04.983045 26121 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.877686 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:04.984534 26092 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:04.962885 26123 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.963299 26132 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.833459 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.985412 25619 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:04.964253 26104 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:04.986060 26104 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.986685 26123 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.987915 26104 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.803085 26117 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:04.989609 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 221209us
I20240426 13:19:04.990131 26117 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.966044 26110 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:04.896981 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 132061us
I20240426 13:19:04.891834 26113 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:04.806124 26122 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.991391 26122 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:04.991904 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.992563 24956 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:04.993484 26122 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.993767 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.994951 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:04.997022 26102 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:04.997409 26102 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:04.997758 26102 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.003026 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:05.003587 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.004240 26117 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.004516 26111 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.004619 26117 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.005117 26117 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.005793 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.006520 26111 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.009742 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:04.881662 26115 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.010572 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:05.011845 26111 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.012290 26111 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.012715 26111 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.013355 26130 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.015426 26130 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.019389 26111 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.021054 26100 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:04.968753 26114 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:04.968765 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:04.805580 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:04.971100 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.009610 26102 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.026748 26102 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.008379 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.023036 24948 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:05.011071 26115 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:04.940670 26107 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.024711 26134 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.025022 26116 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.031867 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:04.975502 26118 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.036358 26117 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.036932 26114 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.053326 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45100.9us
I20240426 13:19:05.021598 26111 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.054680 26134 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:05.054782 26094 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.058730 26094 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:05.058782 24950 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:05.059186 26094 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.054913 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:05.056047 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85013.7us
I20240426 13:19:05.056874 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.068328 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.063841 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.068565 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.063967 26123 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.077443 26123 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.077828 26123 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.079181 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:05.079353 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.086117 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:05.086736 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.086879 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.085034 26133 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.088186 26133 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.084996 26110 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.089735 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:05.090602 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63218.8us
I20240426 13:19:05.094173 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.095098 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.095329 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.097043 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.097250 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.088508 26107 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.099107 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.102414 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.103227 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.088538 26133 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.107007 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.091318 26092 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.112282 26092 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.099963 26117 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.102340 26107 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.119838 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.120293 26115 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.120473 26092 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.120391 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.121261 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.121834 26107 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.089098 26098 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.110356 25615 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.122169 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.153338 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.151949 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.153512 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.152341 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.117486 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.156304 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.158493 25031 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.157912 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.161355 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.161813 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.162856 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.157490 26117 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.161841 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.165253 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.164288 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.166098 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.166764 26098 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.151813 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.171029 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:05.171154 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.171427 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.176658 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.181100 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.181895 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.183096 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.184651 25069 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.185482 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:05.169941 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52575.1us
I20240426 13:19:05.186136 26100 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.186597 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.187124 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.187186 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.187821 26104 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.191507 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.188081 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.187842 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:05.188689 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080013d280 after lost signal to thread 23799
I20240426 13:19:05.193360 26135 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.189188 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.193562 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.194538 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.195216 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.191049 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.190086 26098 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.196174 26104 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.196586 26098 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:05.196969 26098 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:05.196257 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08001754c0 after lost signal to thread 24888
I20240426 13:19:05.198083 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.199718 26125 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.199764 26123 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.201100 25624 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.201773 26123 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:05.202243 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08002b0d40 after lost signal to thread 24980
I20240426 13:19:05.203034 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:05.203295 25126 debug-util.cc:398] Leaking SignalData structure 0x7b0800272720 after lost signal to thread 25055
I20240426 13:19:05.204810 26117 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.204852 26104 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.205013 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.113448 26134 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.206670 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.207787 26134 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.208266 26101 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.209496 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.172238 26133 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.210749 26119 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.213763 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.215905 26129 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.217235 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.219221 26101 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.219333 26093 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.219887 25126 debug-util.cc:398] Leaking SignalData structure 0x7b08002cf0c0 after lost signal to thread 25129
I20240426 13:19:05.220256 26135 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:05.220991 25126 debug-util.cc:398] Leaking SignalData structure 0x7b080009e2a0 after lost signal to thread 25650
I20240426 13:19:05.222211 26107 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.223239 24952 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.226193 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.230022 26117 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.230360 26117 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:05.230691 26117 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.230561 26104 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.237509 26100 raft_consensus.cc:1050] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.239755 26100 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.240198 26117 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.240850 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.242239 26135 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.226099 26094 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.243340 26104 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.243738 26104 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.242417 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.244740 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.244518 26104 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.246683 26117 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.247110 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.247480 26104 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.248210 26133 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.248553 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.249451 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.249054 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.250605 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.259408 25028 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.257059 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.253021 26124 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.257530 25025 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.268652 26124 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.252856 25624 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.261755 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.270200 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.272209 26094 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.272445 25619 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.273538 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.274883 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.269735 26124 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.276172 26109 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.276983 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.278170 26109 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.278234 26110 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.279237 26135 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.280701 26098 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:05.282033 25026 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.284811 26098 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:05.270570 26107 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.286746 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.288700 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.290055 26135 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.291103 26115 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:05.292481 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:05.292603 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.291226 26125 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:19:05.293805 26135 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.294421 26119 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.252667 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.294903 26119 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.295228 26117 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.295565 26125 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.295974 26125 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.294379 26108 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.295711 26119 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.297688 26119 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.299124 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.297313 26125 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.299649 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.300500 26125 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.301487 26129 raft_consensus.cc:1050] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.301261 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.299147 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.302608 26117 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.302894 26101 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240426 13:19:05.291591 26124 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.304750 26135 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:05.305936 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:05.309085 26135 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.309427 26135 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.310194 26135 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.313758 26135 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.315625 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.313959 26129 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.323382 25615 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.325311 26094 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:05.328759 24950 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.331544 26094 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.333865 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.336683 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.337495 25096 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:05.338994 25103 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.342873 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.343856 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.344794 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.345798 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.346869 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.347250 26125 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.347820 26125 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.348608 26094 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.348893 26098 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.349802 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.346897 25589 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.349403 26125 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.351011 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.352607 26117 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.352885 26101 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.353055 26117 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:05.353996 26117 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.355979 26133 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.358808 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.360149 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:05.360903 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.364235 25105 raft_consensus.cc:2379] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.364362 26125 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.366843 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.368573 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.376741 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.381930 26120 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.383155 26097 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.386070 24996 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.383384 26107 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.387794 26107 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:05.388494 25101 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:05.388657 26107 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.395705 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:05.396915 25589 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.397231 25096 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.385555 26119 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.400210 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:05.401202 26099 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.402666 26123 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.403136 26123 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:05.403939 26123 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.396577 26100 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.408861 26119 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:05.410733 26099 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.423230 25097 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:05.440147 24853 catalog_manager.cc:5484] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:05.446139 26107 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.447141 26097 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.448570 24950 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.446821 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:05.454563 26100 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.456223 26100 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:05.458148 26100 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:05.457779 25101 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.473368 24851 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.476925 25615 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.490265 24852 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:05.509581 26112 raft_consensus.cc:1050] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.511970 26112 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.547250 25615 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.552986 26112 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.563880 25023 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.565464 26124 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.565752 25619 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.566442 26124 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.566769 26124 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.569741 24950 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.580518 26121 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:05.581614 26121 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.581929 26121 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.582690 26121 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.586216 26121 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.593179 26132 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.596801 25023 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.598212 26104 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.605584 26104 raft_consensus.cc:1050] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:05.605988 26124 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.608055 26104 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.613466 26124 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.613687 24854 catalog_manager.cc:5484] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:05.633024 25023 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.634346 26132 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.637847 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.638185 24956 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:05.644657 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.645064 25026 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.645443 26112 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.646404 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:05.651007 25619 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.655570 24950 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.657022 26136 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.662333 25615 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.664379 26098 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.668812 25030 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.670087 24949 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.675109 26136 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.677785 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6277899576264e68aa43514e5b5c676e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.677933 24854 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.682184 26104 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.687268 25619 raft_consensus.cc:2932] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.694911 25023 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.698279 24952 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.704141 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:05.702427 25620 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.707754 24949 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.716522 25026 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:05.718809 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 6277899576264e68aa43514e5b5c676e with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:05.724812 26124 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.726094 26124 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:05.726522 26124 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.727998 26124 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.739967 26159 tablet_replica.cc:330] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:05.748867 26159 raft_consensus.cc:2229] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:05.749612 26159 raft_consensus.cc:2258] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:05.750974 25101 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.756578 24851 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.760147 24956 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.760371 25619 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.761933 26112 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.762913 26104 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.769004 26159 ts_tablet_manager.cc:1875] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:05.770622 26104 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.777133 25619 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.794201 24956 raft_consensus.cc:2932] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.795786 26159 ts_tablet_manager.cc:1888] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:05.804332 26159 log.cc:1196] T 6277899576264e68aa43514e5b5c676e P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/6277899576264e68aa43514e5b5c676e
I20240426 13:19:05.798082 26124 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.805929 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 6277899576264e68aa43514e5b5c676e (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:05.817647 25101 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.840247 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9910a5336aa4e3cab11dca4a556fabf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.842514 25026 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.847887 26110 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.850582 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:05.852121 26112 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:05.853863 26112 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:05.853358 24851 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: config changed from index 3 to 4, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.854373 26112 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.857168 26103 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.858479 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:05.859082 25030 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:05.861056 25030 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.864929 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet b9910a5336aa4e3cab11dca4a556fabf with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:05.862490 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:05.867993 26109 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:05.868366 26109 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:05.868738 26109 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:05.881449 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:05.882298 25620 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.882382 26109 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.889909 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:05.891209 25620 raft_consensus.cc:2932] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.893210 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:05.894002 25619 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:05.896456 26109 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:05.903261 26160 tablet_replica.cc:330] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:05.905205 26160 raft_consensus.cc:2229] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:05.905503 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:05.906008 26160 raft_consensus.cc:2258] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:05.907784 25070 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:05.908610 24956 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:05.909659 26121 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:05.909963 26123 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.910429 26121 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:05.911356 26121 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:05.914669 26160 ts_tablet_manager.cc:1875] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:05.917899 24950 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:05.919531 26123 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:05.919837 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c896b58b7766401eac0a7f69574827f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:05.922292 24854 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:05.928411 26112 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.004139 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet c896b58b7766401eac0a7f69574827f1 with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:06.019820 25105 raft_consensus.cc:1261] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.020599 25097 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.023087 26123 consensus_queue.cc:1035] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.024305 26123 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:06.027088 26160 ts_tablet_manager.cc:1888] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:06.027675 26160 log.cc:1196] T b9910a5336aa4e3cab11dca4a556fabf P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/b9910a5336aa4e3cab11dca4a556fabf
I20240426 13:19:06.029448 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet b9910a5336aa4e3cab11dca4a556fabf (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:06.030599 25101 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:06.032519 26098 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:06.040599 24851 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:19:06.047163 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45923.5us
I20240426 13:19:06.053130 24947 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:06.081980 24947 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:06.092685 26166 tablet_replica.cc:330] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
W20240426 13:19:06.099574 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45881.7us
I20240426 13:19:06.102301 26166 raft_consensus.cc:2229] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
W20240426 13:19:06.114255 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52488.7us
I20240426 13:19:06.127043 26099 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20240426 13:19:06.144543 26166 raft_consensus.cc:2258] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240426 13:19:06.156086 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59401.5us
I20240426 13:19:06.187179 26166 ts_tablet_manager.cc:1875] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:06.257426 26166 ts_tablet_manager.cc:1888] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:06.257818 26166 log.cc:1196] T c896b58b7766401eac0a7f69574827f1 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/c896b58b7766401eac0a7f69574827f1
I20240426 13:19:06.259346 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet c896b58b7766401eac0a7f69574827f1 (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:06.356907 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.358153 26109 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.372047 25030 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:06.377082 26121 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:06.506356 26181 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.506820 26181 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.508610 26181 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:06.509614 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.510170 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:06.510185 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:06.512280 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:06.513016 26181 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.513342 26181 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.513583 26181 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.520750 26181 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.522248 26181 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:06.522855 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.523581 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:06.523784 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.530593 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.532351 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:06.533391 26181 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.534034 26181 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:06.534823 26181 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.547057 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:06.556799 26121 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.558270 26121 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.563107 26121 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.563948 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.565342 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:06.566668 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:06.567291 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.568243 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:06.570348 26103 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:06.570708 26103 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.571070 26103 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:06.580511 26103 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.584290 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.584919 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:06.585716 26103 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.589048 26112 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:06.589601 26112 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.590543 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:06.608073 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:06.612100 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.612912 25026 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.613963 25070 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.614256 25070 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:06.614723 26103 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.617380 24956 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.620816 26103 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:06.627768 26119 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.628209 26119 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.631383 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:06.632651 26109 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:06.633417 26109 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:06.634229 26109 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:06.638338 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.639042 25026 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:06.640545 26119 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.641564 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:06.649267 26112 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:06.666720 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:06.667385 25103 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:06.674460 24853 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:06.684571 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:06.686204 26181 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.686616 26181 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.686936 26181 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.697841 26181 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.703752 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.704341 25030 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.710153 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.713415 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:06.714040 25101 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.722888 25101 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:06.724761 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:06.727402 26181 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.727548 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:06.725486 26119 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.728722 26119 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:06.729497 26119 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.745703 24853 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:06.775959 26189 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:06.777247 26189 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.779757 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:06.780406 25030 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:06.789108 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:06.788084 26189 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.788117 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:06.805605 26189 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:06.805994 26189 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:06.806339 26189 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.813710 26189 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.815898 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:06.816502 25026 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:06.821709 25026 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:06.823226 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:06.824092 26189 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:06.823957 26103 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:06.825001 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:06.825196 26103 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:06.826145 26103 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:06.846872 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:07.000401 25030 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:07.001623 26119 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.023224 26103 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:07.023728 26103 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.025718 26103 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.027524 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.028142 25620 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:07.027544 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:07.030539 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:07.031512 26109 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:07.031854 26109 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:07.032178 26109 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.033342 25103 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:07.035144 25030 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:07.037431 26119 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.037771 26103 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.040403 26109 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.042647 26172 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.043145 26172 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.049827 26172 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.051761 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.053772 25101 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:07.055426 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:07.056440 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:07.057228 25025 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:07.057356 26181 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.057813 26181 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.060221 26123 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:07.060532 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.060722 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.061036 26172 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:07.061190 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:07.061336 25620 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:07.061542 26172 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.063318 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.063879 25101 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.070784 24921 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.071080 24921 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:07.071640 26119 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.073745 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:07.074623 26172 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.063488 26181 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.077723 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:07.079810 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:07.084487 25025 raft_consensus.cc:2354] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.089252 26109 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.091631 25589 leader_election.cc:398] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.091989 25589 leader_election.cc:401] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:07.093098 26172 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.094566 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:07.095155 25620 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:07.095306 26119 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:07.102582 24949 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:07.110772 25620 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:07.111886 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:07.114450 26109 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:07.118217 26109 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:07.122454 26109 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.085487 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.130755 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:07.131026 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:07.131536 25023 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:07.137828 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:07.153720 24950 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:07.155480 26189 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:07.160181 25025 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20240426 13:19:07.381871 26195 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.233s user 0.000s sys 0.000s
I20240426 13:19:07.386315 26172 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:07.402413 25096 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:07.450693 25619 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:07.454936 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 283740us
I20240426 13:19:07.461310 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:07.462574 26172 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:07.463050 26172 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:07.463872 26172 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.478096 26179 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:07.478781 26179 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:07.479569 26202 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:07.519127 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:07.605753 26179 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:07.606576 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:07.622138 25615 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Advancing to term 3
I20240426 13:19:07.645623 25101 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:07.646405 25615 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:07.674978 26212 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:07.688021 26205 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:07.719703 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:07.733361 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53838.4us
I20240426 13:19:07.720736 25101 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:07.989724 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:07.996188 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62466.3us
I20240426 13:19:07.998689 24947 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:08.004899 26224 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:08.005302 26224 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:08.005604 26224 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:08.005702 24854 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:08.013028 26224 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.014969 26224 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:08.035773 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.045s sys 0.126s
I20240426 13:19:08.036115 25624 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:08.036375 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.046s sys 0.126s
I20240426 13:19:08.038326 26212 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:08.071712 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:08.075572 25103 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240426 13:19:08.078553 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61597.9us
I20240426 13:19:08.097368 26172 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:08.131182 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.132192 25101 raft_consensus.cc:595] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.132717 26229 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:08.135600 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.136206 25097 raft_consensus.cc:595] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.149912 24852 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:19:08.165073 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 150369us
I20240426 13:19:08.158391 25624 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:08.179548 25097 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:08.180685 25022 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:08.188545 26121 raft_consensus.cc:979] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.191682 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.148339 26109 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20240426 13:19:08.189050 26121 raft_consensus.cc:1067] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.258165 25023 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:08.131120 26211 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:19:08.270009 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66457.3us
I20240426 13:19:08.280381 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:08.280597 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 110361us
W20240426 13:19:08.276211 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73893.8us
W20240426 13:19:08.302747 26234 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20240426 13:19:08.291883 26213 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:08.340989 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64393.9us
I20240426 13:19:08.343602 26247 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:08.381039 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:08.328512 26246 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20240426 13:19:08.416464 26229 raft_consensus.cc:979] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.416934 26229 raft_consensus.cc:1067] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:08.396167 26247 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:08.423687 25620 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:08.424324 25620 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:08.424700 25620 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.441051 25620 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.414335 25615 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:08.456233 25620 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.446614 25615 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:08.475461 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46494.5us
W20240426 13:19:08.487648 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 216965us
I20240426 13:19:08.504118 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:08.504760 25031 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.498029 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.514169 25097 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:08.514513 25097 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:08.515223 25097 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.515859 25615 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.516456 25097 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:08.517681 25615 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.515653 26247 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:08.512689 26123 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:08.567473 26123 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:08.568321 26123 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.569626 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:08.574152 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62035.9us
I20240426 13:19:08.574173 24955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:08.581348 24955 raft_consensus.cc:595] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.587365 25097 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:08.608737 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:08.614356 25031 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:08.641170 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:08.660149 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:08.688361 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:08.688889 25101 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:08.689666 25101 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:08.676769 25022 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:08.696764 25031 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20240426 13:19:08.698724 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50442.1us
I20240426 13:19:08.690387 25101 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.706107 25101 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:08.708711 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55531.6us
I20240426 13:19:08.712556 25102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:08.713145 25102 raft_consensus.cc:595] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:08.714807 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:08.756503 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 48369.7us
W20240426 13:19:08.758169 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016fe60 after lost signal to thread 23799
W20240426 13:19:08.759112 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080016d540 after lost signal to thread 24888
W20240426 13:19:08.760491 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080025dde0 after lost signal to thread 24980
W20240426 13:19:08.761816 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08003660c0 after lost signal to thread 25055
W20240426 13:19:08.763152 25052 debug-util.cc:398] Leaking SignalData structure 0x7b08002a4ce0 after lost signal to thread 25129
I20240426 13:19:08.766374 26172 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:08.766898 26172 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:08.767791 25052 debug-util.cc:398] Leaking SignalData structure 0x7b080032f660 after lost signal to thread 25650
I20240426 13:19:08.796883 25101 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:08.772344 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:08.809863 25024 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:08.767693 26172 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:08.875814 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1), VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2), VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3), VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:19:08.889847 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174729us
I20240426 13:19:08.849449 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:08.894796 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:08.901880 26261 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:08.905643 26172 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:08.907351 26172 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:08.908203 26172 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:08.900869 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63071.8us
W20240426 13:19:08.985321 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78980us
I20240426 13:19:08.990947 26243 raft_consensus.cc:979] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.007002 26109 raft_consensus.cc:979] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.010993 26109 raft_consensus.cc:1067] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.013327 26205 raft_consensus.cc:1067] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:09.019471 25022 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.027599 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44496
I20240426 13:19:09.028340 25615 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.028700 25615 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.039350 25615 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.042286 25615 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.043615 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.044301 25098 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.044799 25098 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:09.045622 25098 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.046958 25098 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.052728 25098 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.053591 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:09.054204 26172 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.054562 26172 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.055140 26172 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.103628 25024 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:09.106316 25024 raft_consensus.cc:595] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:09.126919 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:09.131350 25615 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.137097 25615 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.136757 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:09.139248 24956 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.138242 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:09.140415 25096 raft_consensus.cc:595] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:09.142915 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240426 13:19:09.144610 24852 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.153580 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:09.155416 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:09.218394 26240 raft_consensus.cc:979] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.219018 26240 raft_consensus.cc:1067] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.232475 24956 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.248474 25096 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:09.250388 26172 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240426 13:19:09.266252 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:09.266942 24951 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.267335 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.283876 26228 raft_consensus.cc:979] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.283540 24948 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.284345 26228 raft_consensus.cc:1067] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:09.286115 24954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:09.286624 24954 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:09.287006 24954 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.294111 25030 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:09.296895 24951 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.301457 26123 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 13:19:09.304256 25098 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.304893 25022 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.313103 26250 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:09.314509 26250 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.311196 24954 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.329138 25615 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.330534 25623 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:09.349640 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:09.350071 25024 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:09.352723 24951 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.339080 25615 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:09.350378 24956 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.354635 24956 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.361096 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.362010 25025 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.362237 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.362473 25025 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:09.362931 25026 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.366456 26242 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20240426 13:19:09.372720 25619 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:09.374603 26242 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.379547 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.380196 25104 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.386173 24956 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.395342 24956 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.363655 25025 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.406592 25025 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.414319 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.417788 24954 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.416967 26212 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.428727 24956 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.429203 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:09.395431 26205 consensus_queue.cc:1159] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1590f80bdade41c6a755b8501c55fe16" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:09.432981 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:09.434605 26243 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.434670 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.435287 26243 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.435518 25029 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.436282 26243 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.434311 25026 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:09.453224 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.453804 25104 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:09.454048 25104 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:09.454691 25104 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.405850 24854 catalog_manager.cc:5484] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.444368 25031 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.458604 24854 catalog_manager.cc:5484] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.464879 24956 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.465927 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:09.469033 26250 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.468420 25619 raft_consensus.cc:2932] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.469678 26250 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.470908 26250 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.474584 26172 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:09.506109 25104 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:09.515175 25029 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.523396 25104 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:09.534922 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:09.547672 26181 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:09.548169 26181 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:09.553416 26181 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:09.592749 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 256415e42b6c444387115ab766a5a16c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240426 13:19:09.604398 24839 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55525.2us
I20240426 13:19:09.668256 24852 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:09.705345 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 5, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:09.759604 25104 raft_consensus.cc:1261] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240426 13:19:09.792160 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 153784us
I20240426 13:19:09.799228 24854 catalog_manager.cc:5484] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:09.809404 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 256415e42b6c444387115ab766a5a16c with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:09.815708 26172 consensus_queue.cc:1035] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.825088 24854 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:09.835853 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:09.837730 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.839756 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.841379 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:09.842113 25022 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:09.843253 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:09.843992 26123 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:09.844475 26123 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:09.846072 26123 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.851378 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:09.869210 26123 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.872817 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:09.873406 25022 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.879391 24948 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.880184 26123 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:09.884546 26250 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.886287 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:09.887671 25096 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:09.871304 26295 tablet_replica.cc:330] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
I20240426 13:19:09.889017 26292 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:09.885164 25022 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:09.912144 25025 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:09.913605 26123 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:09.918265 25030 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:09.920224 26255 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:09.928977 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:09.931910 26287 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:09.932406 26287 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:09.933293 26287 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.949069 26295 raft_consensus.cc:2229] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:09.950117 26295 raft_consensus.cc:2258] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:09.952435 26229 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:09.953045 26229 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:09.955267 26229 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:09.956583 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:09.958261 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:09.958917 25030 raft_consensus.cc:2354] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.960575 25070 leader_election.cc:398] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.960932 25070 leader_election.cc:401] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:09.961730 26229 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:09.969462 26295 ts_tablet_manager.cc:1875] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:09.976542 26229 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:09.989490 26295 ts_tablet_manager.cc:1888] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240426 13:19:09.990077 26295 log.cc:1196] T 256415e42b6c444387115ab766a5a16c P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/256415e42b6c444387115ab766a5a16c
I20240426 13:19:09.991875 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 256415e42b6c444387115ab766a5a16c (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:10.001513 25022 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:10.003198 26181 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:10.009047 25104 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:10.012120 26181 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:10.013339 24853 catalog_manager.cc:5484] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:10.440588 25025 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:10.442196 26250 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:10.455636 25104 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:10.457608 26287 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:10.908497 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:10.969285 24956 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:19:10.987167 24920 consensus_peers.cc:485] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c997ccc9a12742c3b149dc9747b2a0ae. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:10.990464 25104 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:10.990464 25025 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:10.992363 26244 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.002318 26255 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.004875 26292 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.019039 25022 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.021293 25030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:11.021896 25030 raft_consensus.cc:595] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:11.010464 25104 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.030915 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9911e0422347463c8781b2574c023744"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:11.031478 25098 raft_consensus.cc:595] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:11.034477 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:11.035112 25104 raft_consensus.cc:595] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:11.043256 26229 raft_consensus.cc:979] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:11.043713 26229 raft_consensus.cc:1067] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:11.045837 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:11.045200 24853 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.047591 24956 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:11.046381 25025 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.048692 25025 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.056070 25025 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.070996 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.071238 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:11.071611 25104 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.071869 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.071961 25104 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.072791 25104 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.073899 25104 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.080170 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:11.081367 25025 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:11.082470 25104 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:11.083951 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:11.084280 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:11.084970 24948 raft_consensus.cc:595] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:11.084947 26311 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.086583 26311 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:11.087680 24920 consensus_peers.cc:485] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53343363b25040c29f4c212b440dc2cc. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:11.088758 25104 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:11.090222 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:11.090284 26307 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.092288 26244 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:11.095448 25025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:11.096150 25025 raft_consensus.cc:595] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:11.098066 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:11.099864 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:11.100656 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:11.087553 26311 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.126197 26244 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.133296 25098 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.134692 25022 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.150013 25104 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:11.175115 25022 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:11.177516 26248 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240426 13:19:11.181125 25069 consensus_peers.cc:485] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 764a1a5c07f94909b2fd69a50de4bedd. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:11.182650 24948 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:11.188505 26267 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:11.195638 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.210300 26244 raft_consensus.cc:979] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:11.210947 26255 raft_consensus.cc:1067] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:11.213351 25623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:11.213933 25623 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.214237 25623 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.239013 24853 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 2 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.248667 26311 raft_consensus.cc:979] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.249174 26311 raft_consensus.cc:1067] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.244819 25623 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.250881 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:11.253582 24956 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.254037 24956 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.255352 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:11.255930 24954 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.256225 24954 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.256896 24954 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.258416 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.259020 25025 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.257930 24954 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.260116 25623 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:11.269012 25025 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:11.280974 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:11.281344 24956 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.282049 26287 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.280941 24954 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:11.283581 26287 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:11.284785 26287 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:11.286314 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.285305 24956 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.286903 25104 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.286321 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.287577 25030 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.287881 25030 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:11.289031 25030 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.290261 25030 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.295622 25104 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.296067 26229 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.298817 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:11.305336 26244 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.305788 26244 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.306622 26244 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.311275 25022 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.315057 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:11.317328 24948 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:11.329857 25030 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:11.341734 24901 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20240426 13:19:11.342902 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240426 13:19:11.363369 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9911e0422347463c8781b2574c023744, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:11.364106 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9911e0422347463c8781b2574c023744, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.408951 26267 raft_consensus.cc:979] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.409431 26267 raft_consensus.cc:1067] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:11.413873 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9911e0422347463c8781b2574c023744"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:11.431824 24853 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.416887 24948 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.455490 24948 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.466836 26323 ts_tablet_manager.cc:902] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:11.475250 24948 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.478063 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.478736 25022 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.497128 25022 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.498667 24948 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.500926 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:11.502640 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:11.511093 25104 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.511437 25104 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.512391 25104 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.513819 25104 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.515607 26253 raft_consensus.cc:979] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:11.516124 26253 raft_consensus.cc:1067] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:11.518143 25096 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:58860
I20240426 13:19:11.518743 25096 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:11.519090 25096 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.524868 26255 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.525444 26255 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:11.526300 26255 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.566710 25098 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.571921 26291 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:11.580655 25096 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.612738 25619 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.636229 26253 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:11.658488 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:11.659168 24954 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.669584 26332 ts_tablet_manager.cc:902] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:11.672082 24954 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:11.673346 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:11.674247 26229 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:11.674724 26229 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:11.675592 26229 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.695578 25104 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:11.698784 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:11.706530 25030 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:11.706977 25030 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:11.721374 25030 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:11.728581 25030 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:11.729436 24853 catalog_manager.cc:5484] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:11.713075 25096 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:11.733331 24851 catalog_manager.cc:5484] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:11.743494 25030 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:11.768783 24948 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.779429 26287 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240426 13:19:11.861213 25022 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:11.861325 25030 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:11.869308 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76510us
I20240426 13:19:11.870509 26342 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:11.871730 26341 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:11.881803 26343 ts_tablet_manager.cc:902] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:11.892318 25097 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:11.912863 26344 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240426 13:19:11.888123 26332 tablet_copy_client.cc:308] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:11.929453 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet 53343363b25040c29f4c212b440dc2cc from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:11.945912 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet 53343363b25040c29f4c212b440dc2cc from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc
I20240426 13:19:11.962152 24966 tablet_copy_source_session.cc:205] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:11.925999 26323 tablet_copy_client.cc:308] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.1:41817
I20240426 13:19:11.984999 26332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343363b25040c29f4c212b440dc2cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:19:12.000450 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87783.1us
W20240426 13:19:12.002002 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80479us
I20240426 13:19:12.005722 24966 tablet_copy_service.cc:140] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Received BeginTabletCopySession request for tablet c997ccc9a12742c3b149dc9747b2a0ae from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:46260)
I20240426 13:19:12.006309 24966 tablet_copy_service.cc:161] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Beginning new tablet copy session on tablet c997ccc9a12742c3b149dc9747b2a0ae from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:46260: session id = 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae
I20240426 13:19:12.014372 24966 tablet_copy_source_session.cc:205] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:12.072718 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997ccc9a12742c3b149dc9747b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20240426 13:19:12.090535 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45430.9us
I20240426 13:19:12.093142 25030 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.098063 26255 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.145236 25097 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.166028 26332 tablet_copy_client.cc:783] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.166673 26332 tablet_copy_client.cc:647] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.179445 26323 tablet_copy_client.cc:783] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.182430 26307 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240426 13:19:12.180076 26323 tablet_copy_client.cc:647] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.190698 26323 tablet_copy_client.cc:523] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.214933 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.220314 26229 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.239074 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.251957 24956 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:12.254038 26354 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:12.272836 26332 tablet_copy_client.cc:523] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.280162 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:19:12.300278 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63772.7us
I20240426 13:19:12.339043 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.358680 26323 tablet_bootstrap.cc:492] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.359499 26323 ts_tablet_manager.cc:1367] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.121s user 0.022s sys 0.003s
I20240426 13:19:12.356315 26343 tablet_copy_client.cc:308] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.3:38089
I20240426 13:19:12.365880 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.362656 26323 raft_consensus.cc:348] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.380142 26323 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.380390 25114 tablet_copy_service.cc:140] P 30c2bb0adfa54bbfa3cb39773d4043c8: Received BeginTabletCopySession request for tablet 764a1a5c07f94909b2fd69a50de4bedd from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:55910)
I20240426 13:19:12.381031 25114 tablet_copy_service.cc:161] P 30c2bb0adfa54bbfa3cb39773d4043c8: Beginning new tablet copy session on tablet 764a1a5c07f94909b2fd69a50de4bedd from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:55910: session id = 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd
I20240426 13:19:12.381901 26323 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.392995 24954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:12.393612 24954 raft_consensus.cc:595] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:12.401533 25114 tablet_copy_source_session.cc:205] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:12.402160 24954 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:12.411176 24954 raft_consensus.cc:595] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:12.412750 26343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764a1a5c07f94909b2fd69a50de4bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:12.402201 26323 ts_tablet_manager.cc:1398] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.042s user 0.004s sys 0.004s
I20240426 13:19:12.425189 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:12.425621 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-c997ccc9a12742c3b149dc9747b2a0ae on tablet c997ccc9a12742c3b149dc9747b2a0ae with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.427431 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:12.431526 25097 raft_consensus.cc:595] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:12.437207 26343 tablet_copy_client.cc:783] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:12.437781 26343 tablet_copy_client.cc:647] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
I20240426 13:19:12.437860 25025 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:12.438705 25025 raft_consensus.cc:595] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:12.440552 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:12.442672 26343 tablet_copy_client.cc:523] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:12.442880 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:12.451213 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:12.473702 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.474834 26332 tablet_bootstrap.cc:492] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.478408 26332 ts_tablet_manager.cc:1367] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.113s user 0.021s sys 0.010s
I20240426 13:19:12.486958 26332 raft_consensus.cc:348] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.489441 26332 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.490154 26332 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.497483 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:12.513144 26332 ts_tablet_manager.cc:1398] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20240426 13:19:12.535058 24966 tablet_copy_service.cc:342] P 1e87e7bb414642aeb72fe0d2ff72cc2e: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc received from {username='slave'} at 127.0.0.1:46260
I20240426 13:19:12.535492 24966 tablet_copy_service.cc:433] P 1e87e7bb414642aeb72fe0d2ff72cc2e: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-53343363b25040c29f4c212b440dc2cc on tablet 53343363b25040c29f4c212b440dc2cc with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.550453 26255 raft_consensus.cc:979] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.550951 26255 raft_consensus.cc:1067] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.553077 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:12.553679 25025 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.554015 25025 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.563433 25623 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240426 13:19:12.601352 25025 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.605789 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:12.606438 24948 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:12.606736 24948 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.607406 24948 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.608409 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.615164 26244 raft_consensus.cc:979] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.615664 26244 raft_consensus.cc:1067] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:12.617190 25029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:12.617760 25029 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.618095 25029 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.625341 25029 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.631649 25029 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:12.634541 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:12.635087 24949 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:12.635392 24949 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.636080 24949 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.635404 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.637825 25098 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.640780 24949 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:12.648262 26349 raft_consensus.cc:979] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:12.648806 26291 raft_consensus.cc:1067] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:12.651715 25098 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:12.653033 24996 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:12.654186 26253 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:12.654824 26253 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.659613 24949 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:12.659525 26253 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.663075 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:12.663991 26343 tablet_bootstrap.cc:492] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:12.664672 26343 ts_tablet_manager.cc:1367] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 0.167s user 0.026s sys 0.000s
I20240426 13:19:12.668424 26343 raft_consensus.cc:348] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.668969 25097 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:58860
I20240426 13:19:12.669798 25097 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.670208 25097 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.669268 26343 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:12.672559 26343 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:12.699427 25025 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:12.702380 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.707182 25096 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:12.718001 25096 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:12.724563 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:12.726387 24996 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:12.731848 26291 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:12.732422 26291 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.733414 26291 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:12.699704 26343 ts_tablet_manager.cc:1398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20240426 13:19:12.768860 25114 tablet_copy_service.cc:342] P 30c2bb0adfa54bbfa3cb39773d4043c8: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd received from {username='slave'} at 127.0.0.1:55910
I20240426 13:19:12.769460 25114 tablet_copy_service.cc:433] P 30c2bb0adfa54bbfa3cb39773d4043c8: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-764a1a5c07f94909b2fd69a50de4bedd on tablet 764a1a5c07f94909b2fd69a50de4bedd with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:12.784540 26345 raft_consensus.cc:979] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:12.790813 26354 raft_consensus.cc:1067] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:12.792191 25097 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.795163 25097 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:12.796018 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:12.796731 25623 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.796018 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:12.797297 25029 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:12.797819 25029 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:12.798554 25029 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.799594 25029 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.801151 24851 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.803673 25623 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:12.805167 24956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:12.805610 25029 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:12.805702 24956 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:12.806896 24956 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.806802 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:12.812052 26267 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:12.812527 26267 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:12.813369 26267 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.820565 24956 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.822396 24956 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:12.829982 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:12.831156 25097 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:12.831481 25097 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:12.831893 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:12.833052 25097 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.837158 25097 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.833911 25623 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:12.850539 25097 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:12.851706 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:12.854488 26255 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:12.854972 26255 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:12.855958 25623 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:12.855748 26255 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:12.881412 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.913084 24852 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.913786 24851 catalog_manager.cc:5484] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:12.973676 25623 raft_consensus.cc:1203] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240426 13:19:13.009521 26307 raft_consensus.cc:1050] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.012151 26307 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.046155 25097 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.046397 25025 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.048746 25098 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.046428 25623 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:13.053332 26255 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.055264 26291 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.057606 26360 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240426 13:19:13.062505 26326 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.163722 25623 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Advancing to term 4
I20240426 13:19:13.165112 26360 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.173703 25029 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.179176 25623 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.188864 24954 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:13.189870 25098 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.191563 26339 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.199744 26291 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.170478 25619 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.218941 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 3 to 4, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } attrs { replace: true } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.266801 26253 raft_consensus.cc:1050] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.269140 26253 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.280615 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.281893 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.291553 25096 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.293385 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.296121 25615 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.301591 26339 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.304802 25029 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.306143 26267 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.308367 26255 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.317173 25097 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.318701 25097 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.318145 25615 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.326228 26255 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240426 13:19:13.331450 25619 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.333926 24956 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:13.334775 26375 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.337829 26349 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.347333 25097 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:13.348775 26349 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.358999 24949 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.370070 25096 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.385110 26307 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 13:19:13.403365 26390 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20240426 13:19:13.404811 24948 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:19:13.408461 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.411211 25615 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:13.411934 26253 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:13.420540 26326 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.436193 25098 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.439157 24949 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } attrs { replace: true } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.476506 26255 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240426 13:19:13.479537 26391 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20240426 13:19:13.500178 25615 raft_consensus.cc:2932] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.510890 24854 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 4 to 5, VOTER 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) evicted. New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.529825 25021 tablet_service.cc:1514] Processing DeleteTablet for tablet c997ccc9a12742c3b149dc9747b2a0ae with delete_type TABLET_DATA_TOMBSTONED (TS 46106ff992984d33bb4a4d8f30162639 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51352
I20240426 13:19:13.536387 24839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c997ccc9a12742c3b149dc9747b2a0ae with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:13.556658 26400 tablet_replica.cc:330] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: stopping tablet replica
I20240426 13:19:13.557930 26400 raft_consensus.cc:2229] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:13.558691 26400 raft_consensus.cc:2258] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:13.561625 26400 ts_tablet_manager.cc:1875] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:13.564533 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:13.583900 26267 raft_consensus.cc:1050] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:13.586126 26267 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.590417 26400 ts_tablet_manager.cc:1888] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240426 13:19:13.591205 26400 log.cc:1196] T c997ccc9a12742c3b149dc9747b2a0ae P 46106ff992984d33bb4a4d8f30162639: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-1-root/wals/c997ccc9a12742c3b149dc9747b2a0ae
I20240426 13:19:13.593334 24840 catalog_manager.cc:4830] TS 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): tablet c997ccc9a12742c3b149dc9747b2a0ae (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:13.604521 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.606376 25031 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.607646 26267 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.607913 24956 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240426 13:19:13.608937 26354 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.612468 26375 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:13.637253 26354 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.639189 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:13.639791 25097 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.667117 25029 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.668390 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:13.668991 24956 raft_consensus.cc:595] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:13.672211 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:13.672780 25098 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.671828 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:13.677510 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:13.678052 24956 raft_consensus.cc:595] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:13.681414 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 44835.5us
I20240426 13:19:13.684617 25619 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.692327 24956 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.696602 25029 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:13.697194 25029 raft_consensus.cc:595] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:13.698292 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:13.699802 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:13.700382 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:13.726263 26354 raft_consensus.cc:979] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.726714 26267 raft_consensus.cc:1067] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.729445 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:13.729966 24949 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.730301 24949 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.730648 26360 raft_consensus.cc:979] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:13.731137 26398 raft_consensus.cc:1067] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 1590f80bdade41c6a755b8501c55fe16 to start an election
I20240426 13:19:13.735158 25615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287"
dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
from {username='slave'} at 127.0.0.1:44512
I20240426 13:19:13.735667 25615 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.736048 25615 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.743815 24949 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.745162 25615 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.747766 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:13.748399 24948 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.748682 24948 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.749984 25615 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:13.750990 24948 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.753187 24948 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.759547 24948 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:13.767320 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:13.768637 26287 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.769214 26287 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:13.770133 26287 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.772773 25098 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.782547 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.783248 25029 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.792171 25619 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:13.793090 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.793390 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:13.793603 25031 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240426 13:19:13.793725 25030 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.794950 26375 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.796113 26354 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:13.793936 25098 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.797161 25098 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:13.799289 25098 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.804297 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.805382 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:13.805322 26354 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.806589 26360 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.807049 26360 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.807901 26360 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.813285 26253 raft_consensus.cc:979] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.813722 26253 raft_consensus.cc:1067] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.814086 25619 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.815603 25029 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:13.815809 25098 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.816926 25025 raft_consensus.cc:2932] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:13.818832 24948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41288
I20240426 13:19:13.823143 24948 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.823467 24948 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.834532 25098 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.836241 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 764a1a5c07f94909b2fd69a50de4bedd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:13.792210 24949 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:13.846768 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: config changed from index 3 to 4, VOTER 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) evicted. New cstate: current_term: 1 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:13.855508 24948 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.872565 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.873262 25031 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.873564 25031 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:13.874281 25031 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.875494 25031 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.879252 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:13.879805 25619 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.883106 24942 tablet_service.cc:1514] Processing DeleteTablet for tablet 764a1a5c07f94909b2fd69a50de4bedd with delete_type TABLET_DATA_TOMBSTONED (TS 1e87e7bb414642aeb72fe0d2ff72cc2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41280
I20240426 13:19:13.886030 25031 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.887683 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:13.888777 26326 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.889237 26326 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.890064 26326 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:13.892122 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.895666 24948 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:13.909585 26410 tablet_replica.cc:330] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: stopping tablet replica
I20240426 13:19:13.926822 26410 raft_consensus.cc:2229] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:13.927666 26410 raft_consensus.cc:2258] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:13.938385 26410 ts_tablet_manager.cc:1875] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:13.953570 26375 raft_consensus.cc:979] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.954125 26267 raft_consensus.cc:1067] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:13.957334 24949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:13.957859 24949 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.958235 24949 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.967476 24949 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.969283 24949 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:13.970139 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:13.970677 25029 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.970764 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:13.971360 25096 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:13.971639 25096 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:13.972343 25096 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.976191 25096 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.976716 25029 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:13.977463 26326 raft_consensus.cc:979] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:13.979753 26307 raft_consensus.cc:1067] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Signalling peer 30c2bb0adfa54bbfa3cb39773d4043c8 to start an election
I20240426 13:19:13.981230 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:13.981909 26307 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:13.982013 25098 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:58866
I20240426 13:19:13.982378 26307 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:13.982532 25098 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:13.982887 25098 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:13.983260 26307 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:13.978345 24853 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:13.989202 24851 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.020531 25029 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:14.039991 24948 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:14.042529 25619 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:14.044605 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:14.046625 26349 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:14.064694 26253 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.068753 24949 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.073583 25619 raft_consensus.cc:2932] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:14.087116 25098 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.127244 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:14.128005 24949 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:14.128394 24949 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:14.129391 24949 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.151338 24949 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:14.150394 25098 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:14.153369 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:14.172520 25030 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:14.164342 24840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53343363b25040c29f4c212b440dc2cc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240426 13:19:14.157184 26410 ts_tablet_manager.cc:1888] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240426 13:19:14.182410 26410 log.cc:1196] T 764a1a5c07f94909b2fd69a50de4bedd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-0-root/wals/764a1a5c07f94909b2fd69a50de4bedd
I20240426 13:19:14.189936 24839 catalog_manager.cc:4830] TS 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): tablet 764a1a5c07f94909b2fd69a50de4bedd (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:14.183738 24853 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index 5 to 6, VOTER 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) evicted. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.148401 25096 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:14.193977 24949 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:14.194207 25030 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:14.195852 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:14.196951 26375 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:14.197399 26375 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:14.200141 26375 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:14.212916 24851 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:14.214227 24851 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:14.220103 24852 catalog_manager.cc:5484] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:14.234455 25095 tablet_service.cc:1514] Processing DeleteTablet for tablet 53343363b25040c29f4c212b440dc2cc with delete_type TABLET_DATA_TOMBSTONED (TS 30c2bb0adfa54bbfa3cb39773d4043c8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58852
I20240426 13:19:14.241365 25096 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:14.248195 26307 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:14.275416 26415 tablet_replica.cc:330] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: stopping tablet replica
W20240426 13:19:14.345556 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56412.6us
I20240426 13:19:14.359506 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.365670 26326 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:14.372874 25029 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.374367 25615 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.378690 24949 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:14.382205 26398 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:14.388015 26338 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:14.433320 26415 raft_consensus.cc:2229] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240426 13:19:14.434429 26415 raft_consensus.cc:2258] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240426 13:19:14.439240 26415 ts_tablet_manager.cc:1875] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240426 13:19:14.891749 26415 ts_tablet_manager.cc:1888] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20240426 13:19:14.892417 26415 log.cc:1196] T 53343363b25040c29f4c212b440dc2cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Deleting WAL directory at /tmp/dist-test-taskC6L1OI/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714137457680803-23796-0/minicluster-data/ts-2-root/wals/53343363b25040c29f4c212b440dc2cc
I20240426 13:19:14.967813 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:15.270458 26354 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.271214 26354 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.273742 26354 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.468458 26354 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.469058 26354 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.471197 26354 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.480535 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.481216 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:15.482453 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:15.483322 26425 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:15.483938 26425 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.484246 26425 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:15.494688 26354 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.495280 26354 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.497370 26354 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.497632 26429 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.506007 26427 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.498122 26429 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.506772 26427 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.508316 26429 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.515092 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.516163 26424 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.516703 26424 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.518671 26424 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:15.525043 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.13096e+06us
I20240426 13:19:15.529631 24840 catalog_manager.cc:4830] TS 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): tablet 53343363b25040c29f4c212b440dc2cc (table test-workload [id=a0cdcee3c79d424793c3372b06b3b129]) successfully deleted
I20240426 13:19:15.538185 26434 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.538851 26434 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.539299 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.540841 26434 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.541960 26427 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.542346 26413 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.542860 26413 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.547438 26291 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.548161 26291 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.550289 26291 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.551287 26425 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.545109 26427 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.554354 26427 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.561743 26413 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.562727 26427 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.565833 26427 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.566396 26427 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.568986 26433 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.569541 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:15.570209 25620 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:15.570333 26430 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.569504 26433 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.570796 26430 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.571236 25025 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:15.573113 26430 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.573783 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.575564 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.580507 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 263) took 1187 ms. Trace:
I20240426 13:19:15.580554 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:15.580641 26425 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.581036 24853 rpcz_store.cc:276] 0426 13:19:14.392787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.577635 (+1184848us) service_pool.cc:225] Handling call
0426 13:19:15.580451 (+ 2816us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
W20240426 13:19:15.583715 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s user 0.007s sys 0.000s
W20240426 13:19:15.584268 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.186s user 0.008s sys 0.000s
I20240426 13:19:15.585088 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 112) took 1189 ms. Trace:
I20240426 13:19:15.585465 24949 rpcz_store.cc:276] 0426 13:19:14.396106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:14.396233 (+ 127us) service_pool.cc:225] Handling call
0426 13:19:14.396540 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:14.396854 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:14.396880 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:14.397569 (+ 689us) log.cc:842] Serialized 26 byte log entry
0426 13:19:15.584682 (+1187113us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:15.584733 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:15.584813 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:15.584862 (+ 49us) raft_consensus.cc:1671] finished
0426 13:19:15.584878 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:15.585010 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":636,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":1186716,"threads_started":1,"wal-append.queue_time_us":963,"wal-append.run_cpu_time_us":1710,"wal-append.run_wall_time_us":1006116}
I20240426 13:19:15.589519 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:15.594318 26429 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:15.594858 26429 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:15.594935 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.596204 26429 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.596776 26427 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.600801 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.608286 26424 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.611002 26424 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.611620 26427 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.612133 26427 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.612888 26424 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.616706 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.547477 26431 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:15.629770 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.22126e+06us
I20240426 13:19:15.622061 26427 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:15.634610 23801 debug-util.cc:398] Leaking SignalData structure 0x7b080000ebe0 after lost signal to thread 24921
I20240426 13:19:15.635267 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.636603 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.636330 26433 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.635519 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.639473 26326 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.639853 26424 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.639984 26326 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.640388 26424 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.641924 26326 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.642583 26424 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.643205 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.646008 26424 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.651178 26424 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.653928 26424 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.648002 26419 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.662318 26419 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.664204 26419 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.627264 26431 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.645627 26433 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.667451 26433 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.672493 26428 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.682227 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.699106 26442 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:15.699924 26433 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.707655 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.703922 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.709815 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.673385 26424 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.710443 25031 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:15.717936 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.718549 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
W20240426 13:19:15.736932 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104293us
I20240426 13:19:15.737538 26354 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:15.755935 26449 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:15.719039 26375 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.715495 26424 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.684880 26428 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:15.805488 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65140.2us
I20240426 13:19:15.805527 26427 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.825613 26432 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.825781 26435 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.826184 26432 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.827047 26446 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:15.827466 26446 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.824054 26354 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.830096 26452 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.811375 26419 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.830585 26452 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:15.831054 24901 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:15.830958 26419 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.832561 26452 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.832839 26419 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.826239 26435 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.829828 26354 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.834462 26426 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.834937 26426 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.835287 26354 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.836562 26435 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.829363 26446 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.836855 26426 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.837363 26453 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.837450 26432 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.765410 26375 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.847069 26427 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.807062 26425 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.811878 26438 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:15.843904 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aca63e93aec8413b83ec63e853c8f403, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.837899 26453 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.856940 26431 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.858685 26424 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.859329 26430 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.859771 26430 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.861220 26460 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.861563 26430 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.861752 26460 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.863570 26453 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.863819 26460 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.845510 26428 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.847407 26375 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:15.848875 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aca63e93aec8413b83ec63e853c8f403, TS 1e87e7bb414642aeb72fe0d2ff72cc2e -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.865272 24851 catalog_manager.cc:5484] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:15.844723 26450 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.849402 26443 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.849910 26437 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.850489 26427 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.848306 26425 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.765223 26449 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.848372 26438 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.867645 26450 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.896222 26460 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.896970 26460 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.897863 26450 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.837527 26444 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.902901 26431 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:15.914707 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 210641us
I20240426 13:19:15.902743 26444 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.910907 26427 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.927949 26438 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.886384 26291 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.928555 26291 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.929267 26425 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.925945 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.878456 26443 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.931208 25620 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:15.903798 26433 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.878585 26437 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.937477 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:15.932462 26442 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.936070 26467 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.938560 26445 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.938380 26433 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.938851 26444 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.936259 26443 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.936740 26419 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.938683 26467 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:15.936283 26453 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.942349 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.943101 25098 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:15.939101 26445 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.944689 26433 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.927953 26427 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.946328 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "256415e42b6c444387115ab766a5a16c"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:15.947731 24949 raft_consensus.cc:595] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:15.944775 26449 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.948921 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.947290 26453 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.940553 26425 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:15.910717 26431 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.947108 26419 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.948783 26438 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.955171 26453 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.955458 26425 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.955710 26472 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.964699 26425 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:15.959209 26443 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.948204 26326 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.940470 26442 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:15.962316 26438 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.962756 26446 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.963167 26449 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.963899 25031 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:47070
I20240426 13:19:15.964210 26467 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.958997 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:15.948042 26460 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:15.969746 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.965546 26428 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.950111 26360 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.972685 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.972611 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.959693 26474 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.978917 25030 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:15.964900 26472 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.975852 25031 raft_consensus.cc:595] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:15.977277 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.966099 26413 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.969170 26455 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.969197 26461 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.960402 26450 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.974072 26419 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:15.979244 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.992192 24949 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:15.981958 25096 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:15.993222 24949 raft_consensus.cc:595] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Received request to transfer leadership to TS 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:15.993520 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.979642 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.969434 26454 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.994038 25097 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:15.982134 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:15.973819 26326 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.952278 26291 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.983225 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.996017 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.994347 25101 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:16.090940 26459 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.091549 26473 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:15.973160 26443 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.091044 26439 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:16.087427 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55227.7us
I20240426 13:19:15.991226 26456 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.981359 26398 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.092186 26473 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:15.996827 26453 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.977468 26446 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:15.983693 26474 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.106148 26466 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:15.995617 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.990985 26437 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:15.990867 26458 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.097599 26464 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:15.978386 26449 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.103332 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.087604 26444 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.004421 26445 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.104581 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:15.981516 26428 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.105746 26440 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:16.096911 26291 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:15.981621 26360 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.091640 26459 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.112581 25030 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:15.992774 26413 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.106632 26466 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.004369 26450 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.004235 26454 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.130882 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.209211 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:16.182360 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 461870us
I20240426 13:19:16.185497 25031 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:16.113179 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.135279 25097 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:16.235862 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:15.992874 26455 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.236904 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.160895 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:16.206471 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 97573.3us
I20240426 13:19:16.246883 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.232635 25619 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:16.247399 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.247972 24948 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:16.249217 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.105332 26467 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.253440 26459 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.253803 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a6a69cdcf2ee4768a20888dd974c6ba7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:16.257696 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a6a69cdcf2ee4768a20888dd974c6ba7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.250002 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.257263 25096 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.258589 26438 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.259881 26455 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.260516 26326 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.202387 26427 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.246338 26432 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.234381 26465 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.261193 26467 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.261348 26432 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.109091 26419 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.105489 26473 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.120368 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.107935 26439 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.203928 26463 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.246584 26431 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.115559 26441 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:16.004364 26461 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.278574 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.289865 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.109370 26398 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.299141 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.300305 26398 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.261646 26465 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.248083 26444 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.306447 24956 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:16.310504 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.262630 26442 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.247905 26291 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.313750 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.323668 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.316277 24952 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:16.328291 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.329414 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.333431 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.334007 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.120870 26453 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.291430 26474 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:16.334666 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96650us
I20240426 13:19:16.161680 26456 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.314368 26433 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:16.334127 26428 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.340987 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49249.5us
I20240426 13:19:16.263473 26464 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.339979 25619 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.314834 26472 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.341827 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.290488 26419 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.333482 26430 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.333729 26354 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.262876 26440 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.348457 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.356982 26453 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.362130 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.363184 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.366500 24954 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.337430 26443 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.368438 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.368626 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.371480 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.342758 26375 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.372622 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.375186 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.370966 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.364708 26428 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:16.364753 26446 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.392602 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.361917 26354 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.393705 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.397220 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.395138 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.393744 24953 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:16.314491 26463 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.398739 24950 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.374447 26375 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.392357 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:16.399734 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.405882 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.263332 26458 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.406966 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.394857 26428 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.408156 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.414144 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.415606 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.416072 24950 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.409091 24956 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.410122 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.410835 26466 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.420284 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.397878 26472 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.411844 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.411896 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.327656 26441 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.406528 26375 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.400596 26435 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.405830 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.417366 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.398480 26354 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.397110 26467 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.411195 26291 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.421547 24950 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.406252 26449 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.418107 26428 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:16.399333 26454 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.428314 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.429328 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.434381 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:16.435930 26441 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.419900 26456 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.422312 26450 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.440789 26468 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.425401 26439 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.439668 26435 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.440564 26466 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240426 13:19:16.411851 26413 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.446607 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.473444 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.474288 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:16.447849 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 88516.7us
I20240426 13:19:16.475360 26440 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.475448 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:16.449173 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 349644us
I20240426 13:19:16.477144 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.477377 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.449626 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.418622 26461 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.450949 25096 raft_consensus.cc:595] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.451817 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.422780 26472 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.479398 26472 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.453900 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.479809 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.479941 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.460834 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.461081 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.480921 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.431406 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.481443 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.444896 26463 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.447152 26444 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.482601 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.482896 26458 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.465734 26425 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.501s user 0.010s sys 0.000s
I20240426 13:19:16.483597 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.483804 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.483482 26425 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.475950 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.487576 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.488314 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.488744 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.486712 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.493366 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:16.509101 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.509905 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:16.764858 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.766618 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:16.511042 26419 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.508951 26485 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.771477 26485 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:16.700515 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d94c0 after lost signal to thread 23799
W20240426 13:19:16.772929 25645 debug-util.cc:398] Leaking SignalData structure 0x7b08002d9ba0 after lost signal to thread 24888
W20240426 13:19:16.774230 25645 debug-util.cc:398] Leaking SignalData structure 0x7b080034dd00 after lost signal to thread 24980
W20240426 13:19:16.775367 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800356d20 after lost signal to thread 25055
I20240426 13:19:16.771901 26485 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.776072 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.465816 26468 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.748450 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.779852 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.780915 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.748932 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.492317 26413 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.781854 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.750388 26435 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.495627 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:16.783653 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 288083us
I20240426 13:19:16.783516 26432 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.784497 26432 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.784931 26432 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.785073 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.785427 26432 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.786190 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:16.786615 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.750859 26375 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.753345 26472 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.753643 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.804955 26465 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:16.808303 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800216940 after lost signal to thread 25129
W20240426 13:19:16.809937 25645 debug-util.cc:398] Leaking SignalData structure 0x7b0800370e60 after lost signal to thread 25650
I20240426 13:19:16.810477 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.812404 26437 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.813161 26437 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.813728 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.814922 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.815155 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.816154 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.816357 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817035 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817385 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.817649 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.818847 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.820305 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.821825 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.787228 26468 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.823724 26460 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:16.824235 26460 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.824676 26460 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.479296 26480 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.757491 25102 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:16.829458 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.758311 26473 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.758788 26425 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.761469 26450 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.835530 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.801801 26427 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.761183 26428 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.763000 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.764180 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.764700 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.770661 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:16.778592 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.872905 24954 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:16.782872 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.508905 26494 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:16.811931 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.832010 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:16.877969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46039.5us
I20240426 13:19:16.879107 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:16.879559 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67725.7us
I20240426 13:19:16.834775 26469 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.834566 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.836122 26450 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:16.762151 26464 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:16.842867 26466 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.850464 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.854537 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.892311 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.856129 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.858238 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.838786 26472 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.862917 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.881423 24948 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:16.894747 24948 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:16.882288 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.895089 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.895432 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:16.865756 26375 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.886569 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.888536 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.883229 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:16.906075 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:16.903784 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.906947 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:16.908078 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.897212 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:16.898401 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.896526 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.882165 26435 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:16.882757 26438 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:16.913801 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.892657 26450 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.891230 26430 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.913856 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.916412 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:16.916867 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.917224 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.890127 26425 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.908867 24948 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:16.912483 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.890549 26480 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:16.920377 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:16.912830 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.974145 24948 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:16.877063 26468 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:16.913964 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.972946 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.976256 26468 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.976326 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.977479 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:16.905113 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 142190us
I20240426 13:19:16.978404 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.979317 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.980258 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.980718 26360 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.981403 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:16.981685 26473 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:16.982117 25624 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:16.982542 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.983600 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.918448 26375 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:16.983856 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.984210 26375 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:16.928758 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:16.984673 26375 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.984968 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:16.928656 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:16.985926 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:16.983767 26453 consensus_queue.cc:1159] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:16.980346 26428 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:16.999433 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:16.991083 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.001183 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.002596 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:16.992976 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80557us
W20240426 13:19:16.993749 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77398.9us
I20240426 13:19:16.918578 26433 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.011823 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.915868 26472 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.013760 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.928555 26438 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.038810 26438 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.041923 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.039605 25029 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:17.044188 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.048939 25618 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:16.988607 26425 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.053859 26425 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.058612 26427 raft_consensus.cc:979] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:17.059049 26427 raft_consensus.cc:1067] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
W20240426 13:19:17.060814 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 551154b9c3494e4da288affc4b88c361, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.061161 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 551154b9c3494e4da288affc4b88c361, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.014497 26494 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.010926 26458 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240426 13:19:17.014377 26413 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.014585 26481 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.014137 26433 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.069651 26433 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:17.001005 26354 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.546s user 0.011s sys 0.006s
I20240426 13:19:17.072387 26354 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.074695 26354 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.011358 26468 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.011091 26426 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.077697 26426 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.078001 26426 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:16.924702 26488 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:17.081876 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 45051.5us
I20240426 13:19:16.924386 26469 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:17.092882 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 163520us
I20240426 13:19:16.903213 26466 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.081936 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68277.6us
I20240426 13:19:17.095870 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.031158 24948 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:17.094431 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.100730 26458 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:17.101140 26458 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:17.101919 26458 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.103130 26458 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:17.014238 26436 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:17.104310 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98782.7us
I20240426 13:19:16.973793 26430 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.108985 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.109261 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.105674 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:16.924073 26419 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.177s
I20240426 13:19:17.108414 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.120122 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.121558 24952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:17.122031 24952 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:17.122426 24952 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.109354 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:17.106333 26439 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240426 13:19:17.110515 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:17.110603 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 213447us
I20240426 13:19:17.128854 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:16.978379 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.113297 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.107766 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.118274 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.119163 24954 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:17.126780 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.100142 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.130821 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.132264 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.132706 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.118834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:17.134603 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.095237 26469 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.126397 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.107159 26456 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.146255 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.146761 26375 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.147231 26497 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:17.147457 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.147775 26497 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.148609 25615 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:17.149363 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:17.149608 26497 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.150131 26448 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:17.140090 26461 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:17.117470 26430 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.126488 26435 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.150420 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.134604 26458 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.175031 26434 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.139854 26469 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.189224 26505 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240426 13:19:17.189996 26466 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.129096 26488 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20240426 13:19:17.196035 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 90447.6us
I20240426 13:19:17.152212 26436 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.148631 26375 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.202440 26505 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.180552 26483 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.206719 24952 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.142026 26432 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.202872 26505 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:17.214247 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87922.1us
I20240426 13:19:17.214965 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.215560 24952 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:17.188673 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 210378us
I20240426 13:19:17.223440 25104 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.223513 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.226503 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.227715 26509 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.228596 26509 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:17.219833 26375 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.208992 26488 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.234488 26432 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.235275 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117063us
I20240426 13:19:17.232719 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.241291 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.209900 26507 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.244784 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.248278 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.249715 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.249682 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.244323 26360 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.252135 25100 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.239681 26375 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.253190 25617 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:17.253990 26375 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.248201 26465 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.252386 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.251722 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.255875 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.225039 26444 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:17.257992 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.258119 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.239636 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.258594 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.259898 25103 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:17.258008 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.263319 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.265476 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:17.235589 26398 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 13:19:17.279063 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.279392 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.274847 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.259406 26375 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.282195 26433 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.284293 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:17.285753 26473 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:17.277915 26438 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:17.286695 26438 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.287147 26438 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:17.286808 26425 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.288882 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.286162 26473 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.289412 25098 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.288890 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.290622 26432 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.290892 26433 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.291690 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.273185 26446 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.293603 26508 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:17.256759 26507 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.297259 26497 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.297677 26497 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.297852 26291 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.258039 26503 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.275262 26424 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:17.291435 26473 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.277805 26436 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240426 13:19:17.305843 26436 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:17.306149 26436 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.292361 26428 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.319291 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.295185 26437 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:17.256865 26505 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.298504 26507 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.299042 26433 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.337107 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.295209 26425 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.343680 26460 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.518s user 0.008s sys 0.000s
I20240426 13:19:17.347292 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.355321 24851 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 277) took 1557 ms. Trace:
W20240426 13:19:17.301148 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69495.6us
I20240426 13:19:17.357501 26450 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.368491 26460 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.306830 26436 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.307533 26497 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:17.373431 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51319.4us
W20240426 13:19:17.374027 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 48990.7us
I20240426 13:19:17.381424 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.348400 26469 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.302421 26424 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.383349 26469 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.323889 26437 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:17.369135 24851 rpcz_store.cc:276] 0426 13:19:15.792598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.792754 (+ 156us) service_pool.cc:225] Handling call
0426 13:19:17.349686 (+1556932us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:15.982821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:16.190808 (+207987us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:16.190838 (+ 30us) write_op.cc:427] Acquired schema lock
0426 13:19:16.190855 (+ 17us) tablet.cc:660] Decoding operations
0426 13:19:16.249948 (+ 59093us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:16.262359 (+ 12411us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:16.263915 (+ 1556us) tablet.cc:683] Acquiring locks for 1 operations
0426 13:19:16.302170 (+ 38255us) tablet.cc:699] Row locks acquired
0426 13:19:16.306309 (+ 4139us) write_op.cc:260] PREPARE: finished
0426 13:19:16.329881 (+ 23572us) write_op.cc:270] Start()
0426 13:19:16.335942 (+ 6061us) write_op.cc:275] Timestamp: P: 1714137556327441 usec, L: 0
0426 13:19:16.337535 (+ 1593us) op_driver.cc:331] REPLICATION: starting
0426 13:19:16.408638 (+ 71103us) log.cc:842] Serialized 530 byte log entry
0426 13:19:16.919686 (+511048us) op_driver.cc:444] REPLICATION: finished
0426 13:19:17.190443 (+270757us) write_op.cc:300] APPLY: starting
0426 13:19:17.190551 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:17.190679 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:17.190814 (+ 135us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:17.191065 (+ 251us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:17.191276 (+ 211us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 13:19:17.191341 (+ 65us) write_op.cc:311] APPLY: finished
0426 13:19:17.191611 (+ 270us) log.cc:842] Serialized 23 byte log entry
0426 13:19:17.191803 (+ 192us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:17.191903 (+ 100us) write_op.cc:446] Released schema lock
0426 13:19:17.191955 (+ 52us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:17.192301 (+ 346us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:17.192337 (+ 36us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":155184,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":267508,"apply.run_cpu_time_us":2116,"apply.run_wall_time_us":2114,"num_ops":1,"prepare.queue_time_us":105066,"prepare.run_cpu_time_us":68686,"prepare.run_wall_time_us":771389,"raft.queue_time_us":171174,"raft.run_cpu_time_us":26845,"raft.run_wall_time_us":197623,"replication_time_us":582061,"thread_start_us":661361,"threads_started":4,"wal-append.queue_time_us":329174}]]}
I20240426 13:19:17.383944 26437 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.384776 25100 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:17.397647 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.399633 25099 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:17.404629 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45056.9us
I20240426 13:19:17.383368 26424 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:17.409420 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.383788 26426 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.430330 26512 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:17.434303 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.412673 26436 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.407053 26509 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.415992 26291 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.432258 26510 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240426 13:19:17.445691 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:17.436985 26485 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.660s user 0.008s sys 0.003s
I20240426 13:19:17.402534 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:17.454028 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 210746us
I20240426 13:19:17.455508 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.456732 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.458948 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.464465 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.465747 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.467617 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.449791 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:17.471467 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.442623 26512 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.469656 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:17.474293 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60923.5us
I20240426 13:19:17.462010 26444 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.475090 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.465384 26458 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.467073 26467 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.467483 26488 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:17.476231 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.467027 26428 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:17.459728 26439 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.498661 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:17.486856 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49305.6us
I20240426 13:19:17.485148 26426 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:17.477026 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76487.2us
I20240426 13:19:17.474572 26465 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.474925 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1102) took 1358 ms. Trace:
I20240426 13:19:17.525774 25026 rpcz_store.cc:276] 0426 13:19:16.111189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.112348 (+ 1159us) service_pool.cc:225] Handling call
0426 13:19:16.116431 (+ 4083us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:16.400326 (+283895us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:19:16.760077 (+359751us) log.cc:842] Serialized 14 byte log entry
0426 13:19:16.961562 (+201485us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:16.997671 (+ 36109us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:17.001163 (+ 3492us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:17.456623 (+455460us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b54002625d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:17.458823 (+ 2200us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:17.469611 (+ 10788us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16019,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":6381,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":11958,"spinlock_wait_cycles":639824128,"thread_start_us":438399,"threads_started":2,"wal-append.queue_time_us":302272}
I20240426 13:19:17.456786 26485 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.530731 26485 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.517704 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.514918 26428 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:17.532497 26428 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:17.532442 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.477762 26433 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.520092 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:17.521158 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:17.521544 26460 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.483351 26513 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.534065 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.534772 25619 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.479593 26456 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.552752 26456 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.553010 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.542781 26454 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.553844 26454 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:17.554067 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.554266 26430 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.542871 26465 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.543756 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.556746 26448 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.557080 26448 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:17.544526 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.557871 26448 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.559330 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:17.560544 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.561291 26442 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.561869 26442 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.562398 26442 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:17.565632 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.569525 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.570575 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.571203 25028 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.571322 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.545527 26428 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.503571 26488 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:17.593874 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.572454 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:17.544620 26438 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.555246 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:18.563351 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 1.00811e+06us
I20240426 13:19:18.564013 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.566843 26505 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.580583 25097 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:17.583623 25070 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.483597 26291 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20240426 13:19:18.568655 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 985068us
I20240426 13:19:18.569912 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:17.594126 25103 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.570734 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 161) took 1014 ms. Trace:
I20240426 13:19:18.571125 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1186) took 1001 ms. Trace:
I20240426 13:19:18.571107 25104 rpcz_store.cc:276] 0426 13:19:17.556746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.569099 (+ 12353us) service_pool.cc:225] Handling call
0426 13:19:18.569804 (+1000705us) spinlock_profiling.cc:243] Waited 876 us on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:18.570679 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2066560}
I20240426 13:19:17.555912 26465 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.571579 25103 rpcz_store.cc:276] 0426 13:19:17.569607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570024 (+ 417us) service_pool.cc:225] Handling call
0426 13:19:17.593963 (+ 23939us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.570899 (+976936us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.571074 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8960}
I20240426 13:19:17.521639 26450 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.522243 26458 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:18.572222 26463 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:17.597247 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:18.573045 26354 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.573519 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1471) took 1004 ms. Trace:
I20240426 13:19:18.573580 26354 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.572487 26465 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.572530 26450 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:17.601547 26438 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:17.595917 26488 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:17.603268 26428 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.558987 26448 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:17.614850 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:17.621023 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:17.631271 25028 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:17.756635 26481 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:17.598248 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:17.665832 26473 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:17.786480 26437 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:17.797433 26430 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:17.885805 26515 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.961025 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1172) took 1131 ms. Trace:
I20240426 13:19:17.985703 26485 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.059396 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 237) took 1174 ms. Trace:
I20240426 13:19:18.460880 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1445) took 1007 ms. Trace:
I20240426 13:19:18.472329 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1164) took 1667 ms. Trace:
I20240426 13:19:18.483134 26413 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.486124 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.490020 26483 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.496923 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 383) took 1196 ms. Trace:
I20240426 13:19:18.528537 26467 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:17.612673 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.533489 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.798797 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 112) took 1194 ms. Trace:
I20240426 13:19:18.799146 25022 rpcz_store.cc:276] 0426 13:19:17.604463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.605631 (+ 1168us) service_pool.cc:225] Handling call
0426 13:19:18.798720 (+1193089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.800145 25617 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:18.800253 26473 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.549057 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.573910 25027 rpcz_store.cc:276] 0426 13:19:17.568820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.568977 (+ 157us) service_pool.cc:225] Handling call
0426 13:19:18.573443 (+1004466us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:17.570504 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.846168 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1476) took 1276 ms. Trace:
I20240426 13:19:18.846508 25030 rpcz_store.cc:276] 0426 13:19:17.570183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570293 (+ 110us) service_pool.cc:225] Handling call
0426 13:19:18.846102 (+1275809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:17.707365 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:18.572571 26431 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.530205 26441 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.874193 26441 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.620569 26481 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.617689 26483 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.897927 26481 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.898468 26483 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.627812 24956 rpcz_store.cc:276] 0426 13:19:17.453283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.453467 (+ 184us) service_pool.cc:225] Handling call
0426 13:19:17.468659 (+ 15192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.482906 (+ 14247us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.482941 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:17.482985 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:17.483012 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.460633 (+977621us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.460697 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.460815 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32395648}
I20240426 13:19:18.636610 26485 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.900429 26485 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:18.642448 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.039s user 0.031s sys 0.075s
W20240426 13:19:18.901043 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.297s user 0.031s sys 0.075s
W20240426 13:19:18.642299 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78341.7us
I20240426 13:19:18.651263 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:18.901803 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1444) took 1457 ms. Trace:
I20240426 13:19:18.575897 26450 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.902112 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.902732 26450 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.903332 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.904156 26450 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.904475 26505 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.905072 26505 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.902153 24950 rpcz_store.cc:276] 0426 13:19:17.444222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.444387 (+ 165us) service_pool.cc:225] Handling call
0426 13:19:17.444792 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:17.445187 (+ 395us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.445210 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:17.603170 (+157960us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:19:17.603495 (+ 325us) log.cc:842] Serialized 26 byte log entry
0426 13:19:18.901386 (+1297891us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:18.901419 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.901479 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:18.901519 (+ 40us) raft_consensus.cc:1671] finished
0426 13:19:18.901532 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.901721 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":330,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4943,"spinlock_wait_cycles":3733760,"thread_start_us":1297543,"threads_started":1,"wal-append.queue_time_us":313100}
I20240426 13:19:18.687144 26493 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.908298 26493 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.909437 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.910204 26465 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.910305 26493 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.687587 26527 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.690323 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.911042 26527 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:18.911471 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 221205us
I20240426 13:19:18.912336 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.913434 26527 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.913543 26507 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.914029 26527 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:18.713445 25099 rpcz_store.cc:276] 0426 13:19:16.804663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.032091 (+227428us) service_pool.cc:225] Handling call
0426 13:19:17.036580 (+ 4489us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.400676 (+364096us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.472016 (+1071340us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.472247 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832620800}
I20240426 13:19:18.715318 25105 rpcz_store.cc:276] 0426 13:19:16.829595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.039456 (+209861us) service_pool.cc:225] Handling call
0426 13:19:17.039698 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.292815 (+253117us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:17.960702 (+667887us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:17.960956 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572681472}
W20240426 13:19:18.717132 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.12813e+06us
I20240426 13:19:18.916728 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.918370 26505 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.581255 26436 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20240426 13:19:18.920004 26436 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.920424 26436 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.921341 26436 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.922698 26436 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.732303 25619 rpcz_store.cc:276] 0426 13:19:16.885340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.885454 (+ 114us) service_pool.cc:225] Handling call
0426 13:19:16.885665 (+ 211us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:17.552475 (+666810us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.059184 (+506709us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.059325 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1492279168}
I20240426 13:19:18.928051 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.738065 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.927145 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:18.929646 26493 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.929805 26505 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.930223 26493 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.932288 26493 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.933131 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:18.933753 25030 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:18.935518 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.936637 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.937220 24955 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:18.937336 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.938091 25105 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:18.939240 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.940337 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.940936 24954 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.943445 26481 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:18.738575 26428 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:18.575937 26354 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.944607 26481 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:18.945792 26446 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:18.946127 26446 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.946870 26446 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.743971 26413 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.772462 26515 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.745716 25618 rpcz_store.cc:276] 0426 13:19:17.300171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.301095 (+ 924us) service_pool.cc:225] Handling call
0426 13:19:17.344232 (+ 43137us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:17.414156 (+ 69924us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:17.416288 (+ 2132us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:17.431932 (+ 15644us) log.cc:842] Serialized 26 byte log entry
0426 13:19:17.562800 (+130868us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b18002cbf60. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:17.612657 (+ 49857us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:17.612709 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.496545 (+883836us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b540062e3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.496606 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:18.496671 (+ 65us) raft_consensus.cc:1671] finished
0426 13:19:18.496688 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:18.496860 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2224,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17314,"spinlock_wait_cycles":107273984,"thread_start_us":49662,"threads_started":1,"wal-append.queue_time_us":184615,"wal-append.run_cpu_time_us":1639,"wal-append.run_wall_time_us":1001692}
I20240426 13:19:18.774853 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.779492 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.793933 26488 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.948004 26446 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:18.949440 26413 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.949440 26515 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.809155 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1475) took 1239 ms. Trace:
I20240426 13:19:18.952210 26488 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:18.953300 25028 rpcz_store.cc:276] 0426 13:19:17.569908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.570395 (+ 487us) service_pool.cc:225] Handling call
0426 13:19:18.809083 (+1238688us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.949819 26438 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.816146 26437 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.817416 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1470) took 1249 ms. Trace:
I20240426 13:19:18.953943 26438 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:18.954275 25026 rpcz_store.cc:276] 0426 13:19:17.568506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.594458 (+ 25952us) service_pool.cc:225] Handling call
0426 13:19:18.817359 (+1222901us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:18.954800 26438 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.956645 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:18.957496 26428 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:18.957852 26428 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.958192 26428 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:18.955845 26438 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.824313 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:18.822839 26503 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:18.962991 25104 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:18.963276 26503 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:18.804010 25029 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:18.828392 26448 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:18.829723 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.20869e+06us
I20240426 13:19:18.963689 26503 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:18.964558 26503 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.965932 26503 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.819520 26467 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.834676 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:18.969518 26467 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.970978 26437 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.836493 24854 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 264) took 2889 ms. Trace:
I20240426 13:19:18.972604 24854 rpcz_store.cc:276] 0426 13:19:15.946901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:15.947044 (+ 143us) service_pool.cc:225] Handling call
0426 13:19:18.836407 (+2889363us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20240426 13:19:18.974272 26449 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:18.974465 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:18.974833 26449 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.971489 26437 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.841609 26497 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:18.977178 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.847998 25616 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:18.978386 26437 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.833981 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1173) took 2004 ms. Trace:
I20240426 13:19:18.871246 26526 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:18.979012 25097 rpcz_store.cc:276] 0426 13:19:16.829940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.043162 (+213222us) service_pool.cc:225] Handling call
0426 13:19:17.043416 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.567852 (+1524436us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.833785 (+265933us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54003e21d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.833931 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1230400768}
I20240426 13:19:18.876482 26441 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.885509 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:18.980608 25623 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:18.889360 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 241) took 1459 ms. Trace:
I20240426 13:19:18.969595 26472 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:18.981449 25617 rpcz_store.cc:276] 0426 13:19:17.446285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.451101 (+ 4816us) service_pool.cc:225] Handling call
0426 13:19:17.456400 (+ 5299us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:18.800609 (+1344209us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:18.889084 (+ 88475us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540063f454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:18.889300 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3089833216}
I20240426 13:19:18.981709 26472 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.917503 26465 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:18.982787 26465 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.983772 25615 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:18.839756 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 270) took 1977 ms. Trace:
I20240426 13:19:18.978587 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 243) took 1357 ms. Trace:
I20240426 13:19:18.867452 26524 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:18.830838 26291 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:18.979578 26526 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.983959 26472 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.999584 26448 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:18.986305 26494 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.000578 26494 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.001219 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.001782 26448 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.002023 25617 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:19.002621 26494 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:18.987095 26413 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240426 13:19:19.003861 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.003968 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.004191 25616 rpcz_store.cc:276] 0426 13:19:17.620815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.628142 (+ 7327us) service_pool.cc:225] Handling call
0426 13:19:18.978509 (+1350367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:19.004539 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1473) took 1435 ms. Trace:
I20240426 13:19:19.002341 24852 rpcz_store.cc:276] 0426 13:19:16.862058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:16.862236 (+ 178us) service_pool.cc:225] Handling call
0426 13:19:18.839687 (+1977451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1265328,"reported_tablets":1}
I20240426 13:19:19.004984 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.004887 25029 rpcz_store.cc:276] 0426 13:19:17.569380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.571093 (+ 1713us) service_pool.cc:225] Handling call
0426 13:19:19.004392 (+1433299us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:19.004485 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2793008896}
I20240426 13:19:19.007638 26494 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.007993 26494 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.008373 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.009035 25623 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:19.008808 26494 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.011411 26494 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.001353 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.012835 26524 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:18.986801 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.014982 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.015331 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.015702 25099 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.015905 25104 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:19.016896 26526 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.017346 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.018108 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.996593 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.019830 24950 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.020170 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.020745 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:19.022006 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.022580 25100 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:18.987422 26465 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:18.991726 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.025368 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.026283 26536 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:19.026758 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.027310 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:19.030216 26481 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:19.010401 26483 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.030555 26494 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.030848 26481 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.031728 26481 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.033161 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:18.976771 26449 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:19.009819 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82770.5us
W20240426 13:19:18.997402 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82511us
I20240426 13:19:19.037745 24954 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:19.038326 24953 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.040207 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.030126 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.042171 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.042912 25070 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.044468 26483 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.044759 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.044859 26413 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.060264 26483 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.060803 26413 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.061587 26413 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.062425 26413 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.062374 26483 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.063311 26507 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.064890 26446 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:19.066100 26446 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.066386 26446 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:19.043562 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44786.9us
I20240426 13:19:19.046509 24955 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:19.049086 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:18.990386 26441 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.058406 26526 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.070063 24948 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:19.051533 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.052557 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.054852 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.043938 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.058880 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.073822 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.017541 26524 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.069761 26441 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.050537 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.067310 26428 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.078640 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:19.081264 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45995.2us
I20240426 13:19:19.109588 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.077271 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.079404 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.111465 26449 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.113467 26494 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.113646 26493 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.092458 26473 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.113770 26449 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.114038 26494 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.092839 26472 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.115056 26472 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.094651 26425 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.115981 26449 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.117753 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.118530 24952 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:19.119067 26413 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:19.117830 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.119472 26413 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.119982 26413 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.094496 26483 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:19:19.120679 26448 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.121176 26448 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.122067 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.123036 26448 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.092800 26438 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.123592 26448 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:19.119415 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.124212 26291 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:19.124558 26291 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.125329 26291 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.113055 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.067122 26446 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.130990 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.093364 26507 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.112864 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82813.9us
I20240426 13:19:19.146307 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.147261 26473 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:19.147720 25588 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.148772 26473 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.070075 26526 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.126564 26291 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:19.128176 26515 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:19.160409 26515 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.160876 26515 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.156664 26431 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.157380 26441 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.115964 26494 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.126752 26448 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.165900 26448 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.126793 26538 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.168275 26538 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.168013 24851 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.159585 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.158464 26526 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.172724 26472 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.140480 26425 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.138841 26488 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.174188 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:19.175894 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55650.7us
I20240426 13:19:19.176504 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.141993 26527 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:19.175540 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44618.1us
I20240426 13:19:19.178601 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.123643 26438 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.149216 26473 raft_consensus.cc:686] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:19.179793 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.173642 26446 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.191073 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:19.185743 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76211us
I20240426 13:19:19.185657 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.188225 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.188490 25100 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.188424 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.193269 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.193434 25023 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.194718 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.195170 25102 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.158430 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.190670 26450 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.200933 26536 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:19.202623 26536 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:19.203809 26536 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.204686 26538 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.216674 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.217293 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.207111 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.180989 26526 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.219537 25100 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.181326 26441 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.241106 25024 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.221185 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.257092 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.228760 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240426 13:19:19.283927 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.220803 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.247447 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.284307 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.285084 25101 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.284137 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.217875 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.245813 25097 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.287312 26465 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.288317 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.288658 26428 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.288930 25022 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.295531 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.296311 25027 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.182179 26527 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.177809 26488 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.192814 26431 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.305002 26431 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.305470 26488 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.306844 26431 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.183537 26425 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.312503 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.313506 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.314898 25023 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.218456 26450 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.323168 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.315562 25022 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.316262 25029 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.250175 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.325515 25099 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.327072 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.328389 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44547.5us
W20240426 13:19:19.328518 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 110741us
I20240426 13:19:19.322232 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.246603 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.212255 26481 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.330715 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 101106us
I20240426 13:19:19.320915 25025 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.334291 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.205827 26467 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.354844 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.359875 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.360775 24950 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:19.241699 26493 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.308975 26505 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.322867 26450 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.374256 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.371011 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.371625 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 57812.6us
I20240426 13:19:19.372457 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.335901 26508 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.408480 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:19.402361 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1469) took 1852 ms (client timeout 1537 ms). Trace:
I20240426 13:19:19.343171 26436 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:19.410604 25025 rpcz_store.cc:269] 0426 13:19:17.550181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:17.550382 (+ 201us) service_pool.cc:225] Handling call
0426 13:19:19.402176 (+1851794us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:19.402287 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3986326272}
W20240426 13:19:19.419317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59267.9us
I20240426 13:19:19.205691 26473 consensus_queue.cc:237] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.402139 26450 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.344024 26448 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:19.421169 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72332.9us
I20240426 13:19:19.360072 26524 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240426 13:19:19.423578 26493 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.365658 26467 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.401664 26465 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240426 13:19:19.401265 26526 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 13:19:19.422660 26494 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.421717 26527 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.442624 26483 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.462263 25027 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.427317 26543 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.415366 26508 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.424117 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.400229 26425 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.458686 26438 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.462755 26494 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.470755 26413 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.426340 26515 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.472002 26494 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.478374 26413 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.479247 26485 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:19.479318 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66598.7us
I20240426 13:19:19.479751 26485 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.480194 26485 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.484283 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.469494 26543 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.487941 26555 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.496085 26555 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.496393 26515 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.496845 26555 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.494740 26465 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.492151 26543 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.498083 26465 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.498075 26555 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.497970 26493 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.501024 26465 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.503549 26465 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.496507 26467 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.504961 26485 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:19.534822 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84515.2us
W20240426 13:19:19.465740 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 272530us
I20240426 13:19:19.540971 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:19.544848 25096 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:19.544991 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.549337 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.552716 25028 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.553325 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.555758 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.558239 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:19.559558 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.562268 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.558439 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.563160 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.566744 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.563171 25104 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.571451 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.574159 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:19.575065 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 127572us
I20240426 13:19:19.576372 26561 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.576550 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.579864 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.581532 26562 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.581956 26562 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.582334 26562 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.577930 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.577951 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.577181 26561 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.593231 26561 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.595382 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.596344 25623 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:19.598381 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:19.600562 26560 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240426 13:19:19.602527 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.603775 26536 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.608235 26467 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.608769 26467 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.610400 26467 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.611256 26413 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.611665 26485 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.612418 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.613073 26526 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.612584 26505 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20240426 13:19:19.621208 26505 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.621611 26505 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.622675 26505 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.626391 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:19.627908 24977 debug-util.cc:398] Leaking SignalData structure 0x7b0800200240 after lost signal to thread 23799
I20240426 13:19:19.628566 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:19.629299 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3aa0 after lost signal to thread 24888
I20240426 13:19:19.628931 26505 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.629738 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf480 after lost signal to thread 24901
W20240426 13:19:19.632093 24977 debug-util.cc:398] Leaking SignalData structure 0x7b080026aba0 after lost signal to thread 24980
W20240426 13:19:19.628628 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 50835.8us
I20240426 13:19:19.614413 26446 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:19.634732 26436 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.633929 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08003229a0 after lost signal to thread 25055
I20240426 13:19:19.634851 26446 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.637471 26446 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.611989 26413 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.638211 26413 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.638567 26413 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:19.639757 24977 debug-util.cc:398] Leaking SignalData structure 0x7b0800165aa0 after lost signal to thread 25129
I20240426 13:19:19.641722 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:19.613536 26438 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:19.645716 24977 debug-util.cc:398] Leaking SignalData structure 0x7b08003092a0 after lost signal to thread 25650
I20240426 13:19:19.645341 26438 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.648687 25104 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:19.650363 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61766.2us
I20240426 13:19:19.651108 26441 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.634240 26431 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.656574 26441 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.613003 26448 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.634550 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.649657 26438 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.669628 26524 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.673079 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.674325 26503 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240426 13:19:19.674532 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 85009.7us
I20240426 13:19:19.675444 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:19.686390 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52020.8us
I20240426 13:19:19.689433 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.690434 25096 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:19.690981 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.692070 26508 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.692330 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:19.658227 26441 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.689522 26450 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.716454 25030 raft_consensus.cc:2354] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.700155 26354 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:19.700253 26291 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.525s user 0.008s sys 0.008s
I20240426 13:19:19.701011 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.701928 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.704349 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.700387 26481 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.723750 25026 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 3. Ops: []
I20240426 13:19:19.724314 26450 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:19.726361 26508 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:19.728595 26508 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:19.729068 26413 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.729683 26508 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.731091 26413 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:19.733564 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58293.6us
I20240426 13:19:19.734472 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.735364 26431 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:19.735908 26431 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.736856 26441 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.736879 26431 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.740727 26551 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:19.741814 26551 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.742280 26551 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.743889 26483 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.743891 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.743325 26551 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.739605 25070 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.748478 25070 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:19.725802 26562 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:19.749315 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 499552c0877e46c5af41389ff1be3152, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:19.749677 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 499552c0877e46c5af41389ff1be3152, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.749524 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.750113 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.752643 26562 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.753530 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.753993 25615 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.756378 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:19.758486 26551 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.764137 25070 leader_election.cc:398] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.764485 25070 leader_election.cc:401] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:19.765045 26535 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:19.765619 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.768110 26526 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.772651 25615 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:19.773852 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:19.774669 26465 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.771804 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.776190 26465 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.776794 24854 catalog_manager.cc:5484] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.777375 26465 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.775216 24852 catalog_manager.cc:5484] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.786247 26526 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.786592 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.786617 26526 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:19.787958 26562 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:19.780349 26483 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.788228 26526 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.795543 26553 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.796062 26553 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.797153 26553 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.800993 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.803051 25615 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:19.803229 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:19.804591 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.791486 26431 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.808741 26553 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.810009 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.810057 24951 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.812777 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:19.813591 26535 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.813943 26535 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.814505 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.797631 26467 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.825574 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.816529 26483 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.818634 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.819788 24952 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:19.820807 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.828261 25623 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:19.827941 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.814244 26535 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.834883 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.835726 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.836930 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.838371 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.838537 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:19.839246 25027 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:19.846905 26524 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.847399 26524 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.850158 26505 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.858860 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59992.7us
I20240426 13:19:19.859416 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.860756 26354 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:19.861809 26354 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:19.862124 26354 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.863026 26354 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.867336 26354 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.869266 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:19.865917 24956 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:19.865861 25588 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.877159 26526 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:19.878633 26483 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.878579 24852 catalog_manager.cc:5484] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.848321 26524 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.884912 26524 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:19.886170 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86712.5us
I20240426 13:19:19.890987 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.896826 26424 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.897449 26424 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:19.898126 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.899009 26568 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.900027 26568 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.900238 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:19.901526 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.903528 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.905781 26505 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.909471 26515 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:19.910112 26483 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.909992 26515 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.911266 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.911898 26515 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.915319 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:19.915059 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:19.918462 24955 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:19.904175 26526 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:19.920953 26431 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:19.922878 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:19.923151 26535 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.923741 26515 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.923449 26467 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.924188 26515 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:19.924693 26515 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.925408 26535 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:19.930789 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.931380 24955 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.924505 26467 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240426 13:19:19.917595 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.936069 26483 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.936318 26551 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.936818 26551 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.938604 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.939316 26507 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.939443 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.939949 26291 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.940941 26507 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:19.941339 26507 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:19.937728 26551 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.941687 26507 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.939339 24955 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:19.943248 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:19.952054 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.942174 26483 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240426 13:19:19.942512 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.943007 26551 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.956840 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.958097 26291 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.958658 26291 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.947155 26515 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.935088 24852 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.959899 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:19.961474 26535 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:19.961519 26467 raft_consensus.cc:2741] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 13:19:19.962121 26535 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:19.962433 26467 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:19.962982 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.963816 26553 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:19.963933 26467 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.952390 24854 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.964231 26553 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:19.965046 24854 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:19.968492 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:19.969272 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:19.970310 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:19.970877 25098 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.968328 26291 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240426 13:19:19.971241 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:19.965154 26553 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:19.972380 26568 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.972797 26553 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.973037 26508 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:19.979173 25098 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:19.981743 26527 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.983498 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:19.961846 24852 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:19.986625 26524 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:19.990386 26467 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:19.990842 26526 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:19.997874 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:19.998481 24948 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:19.963400 26535 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.010489 24954 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:20.010380 26503 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.011957 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:20.012092 26563 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:20.013480 26563 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:20.017423 26467 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.019491 26467 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.020666 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.022118 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.024695 26515 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:20.025394 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.026355 25022 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.025497 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.032820 24851 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:20.037540 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:20.039680 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.041642 26551 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:19.999523 26524 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.045336 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.058646 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:20.069063 25028 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.081821 26515 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.070531 26431 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:20.090581 24954 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:20.104454 26424 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:20.105599 26424 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.106009 26424 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:20.106810 26424 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.107826 24854 catalog_manager.cc:5484] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:20.111974 26424 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.134109 25615 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:20.111192 26431 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:20.141036 25022 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.145386 26467 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:20.150367 26524 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:20.152115 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:20.152433 26291 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.153021 25615 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:20.152972 26291 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.146530 26507 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.154832 26291 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.155176 26507 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:20.155767 25098 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:20.156646 26575 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20240426 13:19:20.157013 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.157388 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.157948 25100 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.160961 24954 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.159461 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.168903 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.170178 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.173173 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.172477 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.177141 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:20.143674 26508 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240426 13:19:20.167374 26527 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:20.170578 26483 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.165004 26524 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240426 13:19:20.207391 24956 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.210143 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.192538 26505 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.202963 26483 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.175978 26535 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240426 13:19:20.215600 26505 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.216136 26561 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:20.224570 26483 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.179175 26431 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.248759 24956 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:20.237828 26505 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:20.280753 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146499us
I20240426 13:19:20.294382 26553 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.306869 26553 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.308043 26553 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.309854 26553 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:20.295425 26579 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20240426 13:19:20.300941 26551 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:20.261385 26561 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
W20240426 13:19:20.328013 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 125231us
I20240426 13:19:20.330494 25100 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:20.331399 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 111849us
I20240426 13:19:20.333686 25023 raft_consensus.cc:1261] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:20.335566 26481 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:20.338186 25023 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.347865 25098 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:20.359828 26576 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20240426 13:19:20.368362 26562 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.389679 26291 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.390396 26291 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.416443 25623 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:20.427742 26291 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.429174 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.431586 25099 raft_consensus.cc:595] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.432770 24956 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:20.432708 26467 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:20.435047 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.435431 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.435748 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.436143 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.438377 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.395606 26561 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:19:20.444800 26483 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.449611 26438 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.450297 26413 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.453847 26515 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:20.486827 25098 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:20.492362 25028 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:20.495779 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 154215us
I20240426 13:19:20.502036 25096 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.484156 26413 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.502667 25096 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:20.489063 26467 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.515185 26413 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.494544 26526 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240426 13:19:20.489972 26438 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:20.539752 26597 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240426 13:19:20.541173 26505 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.544670 26505 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:20.552467 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.554070 26483 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.556056 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.556715 25098 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.564267 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.564970 26465 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.565160 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.565513 26465 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.567449 26465 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.569912 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:20.570755 25617 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:20.532100 26467 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.578382 26467 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.586520 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.587579 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.588272 25031 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.592602 26438 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.593930 25025 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.597805 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.598639 26553 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.599004 26553 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.599336 26553 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:20.600917 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.601557 25099 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.602994 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.603787 25102 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.604514 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.605212 25105 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.606724 24948 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:20.613466 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:20.614145 25103 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.617753 25028 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:20.618067 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.619865 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.620205 25098 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:20.626901 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:20.631812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.632479 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:20.636600 24852 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:20.630002 26568 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:20.643998 26438 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:20.668311 26438 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:20.653411 26562 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:20.672286 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.657898 26481 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.668167 26589 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240426 13:19:20.670465 26568 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:20.688359 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 63045.5us
I20240426 13:19:20.688796 25028 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:20.691591 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:20.677569 26562 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:20.701352 26438 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.703505 26553 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.704890 24948 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:20.705473 24948 raft_consensus.cc:595] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.706043 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.687510 26568 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.697261 26467 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:20.683840 26413 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.698657 26562 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:20.683743 26481 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.722229 26553 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.703958 26600 consensus_queue.cc:1035] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240426 13:19:20.725642 26561 raft_consensus.cc:979] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:20.726117 26561 raft_consensus.cc:1067] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:20.728461 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.729135 25096 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:20.730965 26481 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.731359 26467 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.732174 26467 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:20.738078 24953 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:20.742219 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:20.743562 24950 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:20.748348 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.749317 26291 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:20.749639 26291 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:20.749840 26291 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:20.750890 24950 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:20.751659 24950 raft_consensus.cc:595] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:20.753605 25028 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:20.754966 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:20.760162 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81360.5us
I20240426 13:19:20.763471 24954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:20.761818 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:20.764047 24954 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:20.772470 24954 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.782478 26441 raft_consensus.cc:979] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.764372 26526 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:20.810659 24954 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.821655 26610 raft_consensus.cc:979] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.822202 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:20.823017 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:20.823732 26527 raft_consensus.cc:979] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.801496 26291 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:20.855813 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62532.6us
I20240426 13:19:20.855849 24954 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:20.857334 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61162us
I20240426 13:19:20.873209 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.873888 24954 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:20.794445 26611 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240426 13:19:20.874696 26551 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.875144 26551 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.877025 26551 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.878490 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:20.879051 25096 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:20.878886 26562 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:20.879438 25096 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:20.875411 26610 raft_consensus.cc:1067] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.803642 26553 raft_consensus.cc:1067] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.890733 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.892442 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:20.893162 25023 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.896670 26291 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.907557 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:20.927561 24948 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:20.906229 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:20.939721 24954 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:20.930963 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:20.945514 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:20.922852 26563 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:19:20.946456 25025 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:20.961776 26291 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:20.902689 25096 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.009497 26588 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:21.009577 25028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:21.010242 25028 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.010584 25028 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.010656 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:20.871107 26526 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:20.906116 26431 raft_consensus.cc:1067] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:20.972481 26413 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:20.994765 26568 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.003679 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.012199 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:21.009974 26588 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:21.026082 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:21.109267 24950 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.109714 25025 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.116870 25025 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.109948 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.122021 25100 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:21.124752 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98743.6us
I20240426 13:19:21.124472 26291 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.126257 25103 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.135130 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132647us
I20240426 13:19:21.136561 26291 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.171779 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:21.134343 26614 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.177799 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.269479 25104 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:21.177830 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56003us
I20240426 13:19:21.121421 26481 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:21.171587 26473 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.252413 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.262192 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 235921us
I20240426 13:19:21.278321 25028 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.299324 25028 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.266173 25023 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.267562 25024 raft_consensus.cc:2354] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:21.129714 26562 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.275023 25104 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.131290 26551 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.280756 25025 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.310463 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:21.287264 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.010s sys 0.017s
I20240426 13:19:21.331400 25025 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:21.340056 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.011s sys 0.018s
I20240426 13:19:21.312291 26291 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:21.276543 26515 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.311045 26551 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:21.316504 26481 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.370373 26481 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.357065 26610 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.334295 26483 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:21.379326 26610 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.312084 26614 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.351083 25026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:21.385613 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.359709 26551 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:21.395072 25102 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:21.396513 25102 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:21.311683 26473 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.352660 26515 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.294449 26588 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.404615 26524 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.430833 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.426785 26413 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.435290 25104 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.413518 25026 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:21.413432 25102 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.338275 26526 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.410521 26562 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.442348 25026 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.443756 25102 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:21.444185 26619 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
I20240426 13:19:21.446105 26610 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.442404 26413 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.387104 26483 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.473594 26483 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.457187 26473 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.458369 25100 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.469143 26622 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.335s user 0.000s sys 0.010s
I20240426 13:19:21.478197 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.479830 24996 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.470161 26614 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:21.471807 26620 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20240426 13:19:21.478964 25100 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.494563 25102 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.433410 26568 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.497628 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.505712 24996 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.498468 26617 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.510511 25104 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.513273 24996 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.473603 26413 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.454499 26524 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:21.526582 26524 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.523295 26450 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:21.529227 26450 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.530063 26450 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.531103 26588 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.493023 26562 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.532167 26618 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:21.532691 26618 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.533668 26562 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.514951 26617 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:21.533538 26618 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.534740 26617 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.510068 26431 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.510790 26483 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:21.559248 25026 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.548617 26562 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.551169 26431 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.568113 26431 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.551105 26483 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.588582 25096 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.560266 26515 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.591320 24854 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:21.606983 26483 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.637712 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:21.642781 25105 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:21.644582 25096 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.643123 25105 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:21.595922 26562 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.661469 25105 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.682857 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.661530 25026 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.702318 26291 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.702988 26291 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.711581 26291 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.712612 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:21.713706 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 426916us
I20240426 13:19:21.710287 26588 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.710906 26473 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.743834 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.751205 26602 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.777420 26625 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.777899 26625 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.775589 26602 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.756587 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.766952 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.770522 25617 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:21.774124 25026 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:21.781070 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.785636 25104 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.765816 26595 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.790616 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.792306 25102 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:21.779914 26625 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.805496 26467 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.806600 26467 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.765689 26588 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.770154 26473 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.807651 26602 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.803244 26625 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.821362 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.013s sys 0.025s
W20240426 13:19:21.821749 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.518s user 0.013s sys 0.025s
W20240426 13:19:21.831073 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.012s sys 0.053s
W20240426 13:19:21.831612 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.012s sys 0.053s
I20240426 13:19:21.843677 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:21.846285 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.004s sys 0.029s
W20240426 13:19:21.847096 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s user 0.004s sys 0.029s
I20240426 13:19:21.850423 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.819212 26562 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.852356 26467 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:21.862252 26291 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:21.862811 26291 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.860524 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:21.796705 26595 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.864269 25615 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.864480 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.867240 24948 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.869741 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.870865 24948 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:21.856153 26624 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.871155 26291 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.821909 26625 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.856338 26441 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.872696 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53337.6us
I20240426 13:19:21.874966 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.876498 24955 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:21.876765 24955 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:21.862717 26568 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.865136 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.024s sys 0.027s
I20240426 13:19:21.878834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:21.880230 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.025s sys 0.027s
I20240426 13:19:21.863410 26632 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.865325 26635 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:21.880548 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 570133us
I20240426 13:19:21.880465 25102 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.878746 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.881830 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.889899 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:21.890331 25028 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:21.890426 24951 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.882347 24956 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:21.890853 25025 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:21.882808 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.883562 26534 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20240426 13:19:21.893764 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.894362 25104 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.892560 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.897629 24921 leader_election.cc:398] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:21.897980 24921 leader_election.cc:401] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:21.898430 25623 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:21.899909 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:21.900476 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:21.900859 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:21.902774 25615 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:21.884745 24955 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.903604 24955 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.885561 26632 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:21.906122 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.898630 25616 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:21.883934 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.906708 25027 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:21.909415 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.909163 24948 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:21.910154 24956 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:21.911760 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.901947 26611 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:21.917678 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:21.914295 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:21.879554 26441 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.876125 26624 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:21.917229 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:21.918561 24948 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:21.886077 26568 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.920743 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.943411 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.945241 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.908888 26632 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:21.954916 25025 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:21.947093 25027 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:21.947741 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.998157 25030 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:21.947903 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.912012 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:21.948402 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.948513 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:21.948637 26441 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:21.949240 25024 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:21.914563 26588 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.919981 26618 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:22.015935 26630 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:21.993096 24955 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.993562 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74455.6us
I20240426 13:19:21.986968 26291 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.041975 26630 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:21.915349 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:21.995466 26641 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240426 13:19:21.995362 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:21.919881 26473 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:21.986562 24950 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.053807 25025 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:22.058293 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.058424 26595 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.059104 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.059726 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.001215 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:22.042497 26291 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:21.914702 26640 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.902853 26635 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.047618 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.047899 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.053084 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:22.111173 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:22.111558 25031 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:22.107966 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 60430.1us
I20240426 13:19:22.112988 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:22.107244 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65027.5us
I20240426 13:19:21.997421 26617 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.000849 26553 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:21.922156 26611 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.052757 26505 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.192719 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240426 13:19:22.198159 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197061us
I20240426 13:19:22.053913 25616 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:22.198540 26291 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.209498 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.210620 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.212654 25025 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:22.111059 26640 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.212512 25104 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:22.220518 24954 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:22.225617 26291 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:22.225888 24996 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:22.191329 26467 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.208544 26553 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.229691 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.226725 25096 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:22.192354 26441 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240426 13:19:22.223107 26640 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.232333 26467 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.285637 26467 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.226887 26526 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.286276 26526 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.288112 26526 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.239290 26553 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.292196 26467 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.294142 26467 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.263733 26618 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:22.224258 26505 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.232583 26625 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:22.259668 26646 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.305s user 0.000s sys 0.000s
I20240426 13:19:22.295524 26553 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.296945 26551 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.871s user 0.011s sys 0.000s
W20240426 13:19:22.320219 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 274574us
W20240426 13:19:22.329571 26649 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.277s user 0.001s sys 0.000s
W20240426 13:19:22.329705 26642 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.328s user 0.001s sys 0.000s
I20240426 13:19:22.330684 26630 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:22.331136 26624 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.333068 24951 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:22.351402 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 137551us
I20240426 13:19:22.341287 25024 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240426 13:19:22.345834 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 155765us
W20240426 13:19:22.360255 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 132747us
I20240426 13:19:22.328362 26508 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.363761 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:22.383422 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:22.387830 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:22.392385 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.393311 24948 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:22.396812 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.398257 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.362130 26568 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.404992 26626 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:22.405198 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:22.405495 26654 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.406245 26635 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.407408 24952 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:22.413771 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:22.347577 26551 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.422009 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:22.414620 24955 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.400799 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.392565 26508 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.423266 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 197446us
I20240426 13:19:22.451982 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:22.485314 25029 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:22.486516 26508 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:22.488099 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 222117us
I20240426 13:19:22.489045 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:22.437884 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.492153 25617 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:22.493580 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:22.499632 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.500728 25103 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:22.501253 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:22.501868 25096 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:22.504727 25096 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:22.513659 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88292.7us
I20240426 13:19:22.423285 26654 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.554931 26654 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.296082 26618 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.583714 26669 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:22.586655 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 93144.2us
W20240426 13:19:22.437129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 137296us
W20240426 13:19:22.598584 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 234919us
I20240426 13:19:22.576277 26481 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.415019 26586 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:22.628561 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 138179us
I20240426 13:19:22.594161 26625 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.591213 26611 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.627701 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:22.584215 26669 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:22.593000 26508 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.652763 26611 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.655743 26663 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:22.656139 26526 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.630944 26586 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.656388 26656 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:22.656780 26648 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.392s user 0.000s sys 0.001s
I20240426 13:19:22.657066 26601 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:22.657377 26659 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:22.657761 26623 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20240426 13:19:22.656201 26660 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:22.653829 26669 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:22.630250 26553 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.658154 26505 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.650532 26635 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.631770 26481 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.690760 26481 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.667963 26678 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.691540 26678 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.691953 26678 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.684515 26659 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.694850 26659 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.669656 25105 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:22.670192 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 71797.5us
I20240426 13:19:22.671064 26665 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.663772 25616 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.700502 26665 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.702661 26665 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.631986 26625 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.703266 26625 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.705088 26617 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:22.678826 26645 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.762s user 0.005s sys 0.003s
I20240426 13:19:22.705607 26617 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.679226 26551 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.677460 26586 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.683553 26601 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.707433 26601 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.678102 26526 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:22.678324 26483 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.036s user 0.011s sys 0.000s
I20240426 13:19:22.680316 26656 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.658435 26611 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.661470 26664 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.687021 26660 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.688336 26635 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.722769 26586 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.668548 26508 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.877642 25105 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:22.881110 26661 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:22.881335 26562 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:22.881865 26661 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.882012 26562 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.881188 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:22.729620 26656 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.896224 26551 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:22.896728 26551 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:22.897111 26551 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:22.904525 26562 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.913585 26508 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:22.887440 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1255) took 1261 ms. Trace:
I20240426 13:19:22.915241 25105 rpcz_store.cc:276] 0426 13:19:21.619294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.619429 (+ 135us) service_pool.cc:225] Handling call
0426 13:19:22.654775 (+1035346us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b58003101b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:19:22.880681 (+225906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139556224}
I20240426 13:19:22.729933 26483 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.152951 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.008s sys 0.002s
W20240426 13:19:23.485831 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.859s user 0.010s sys 0.007s
W20240426 13:19:23.012367 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 423422us
W20240426 13:19:23.183581 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.005s sys 0.030s
I20240426 13:19:22.925295 25622 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:23.152743 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.026s sys 0.021s
W20240426 13:19:23.495303 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s user 0.026s sys 0.029s
W20240426 13:19:23.487675 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s user 0.008s sys 0.041s
I20240426 13:19:22.856907 26611 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:23.270577 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.008s sys 0.013s
W20240426 13:19:23.498123 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.016s sys 0.033s
W20240426 13:19:23.402556 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.023s sys 0.017s
W20240426 13:19:23.499322 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s user 0.023s sys 0.019s
W20240426 13:19:23.493815 24948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.006s sys 0.012s
W20240426 13:19:23.500785 24948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s user 0.006s sys 0.012s
W20240426 13:19:22.918969 26632 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.926s user 0.009s sys 0.000s
I20240426 13:19:23.500604 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.501605 25027 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:23.242197 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.013s sys 0.063s
W20240426 13:19:23.239195 26640 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.954s user 0.007s sys 0.004s
I20240426 13:19:23.504364 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 449) took 1638 ms. Trace:
W20240426 13:19:23.481361 26656 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.589s user 0.009s sys 0.000s
W20240426 13:19:23.423310 26617 consensus_meta.cc:287] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.717s user 0.009s sys 0.003s
I20240426 13:19:23.504725 25623 rpcz_store.cc:276] 0426 13:19:21.859768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.859902 (+ 134us) service_pool.cc:225] Handling call
0426 13:19:21.860162 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:21.898782 (+ 38620us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:23.494606 (+1595824us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:23.497799 (+ 3193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83563392}
W20240426 13:19:23.141587 25026 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.015s sys 0.027s
W20240426 13:19:23.489593 26618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s user 0.013s sys 0.005s
W20240426 13:19:23.497550 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.006s sys 0.005s
I20240426 13:19:23.509886 25022 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
W20240426 13:19:23.428153 26654 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.873s user 0.012s sys 0.000s
I20240426 13:19:22.922345 25624 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:23.204550 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.069s sys 0.133s
I20240426 13:19:23.512284 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:23.564177 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.069s sys 0.135s
I20240426 13:19:23.512593 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:23.514226 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.115s user 0.021s sys 0.064s
W20240426 13:19:23.564512 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.04993e+06us
I20240426 13:19:23.564756 25025 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:22.886185 26664 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:22.920466 26678 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.287484 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.917755 26586 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:22.916088 26635 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.515636 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:22.922070 26660 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.557361 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.484979 26562 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:23.556344 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.903s user 0.006s sys 0.005s
W20240426 13:19:23.560024 26618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s user 0.013s sys 0.006s
I20240426 13:19:23.371758 26665 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:23.384109 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.648s (SENT) [suppressed 2 similar messages]
W20240426 13:19:23.557101 25026 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.180s user 0.019s sys 0.039s
I20240426 13:19:23.569469 25027 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:23.173694 26601 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.565524 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:23.570030 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:23.204200 26676 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.000s sys 0.010s
I20240426 13:19:23.099735 26661 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:22.965708 26625 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:22.922991 26669 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.584832 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 941505us
W20240426 13:19:23.490295 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.008s sys 0.003s
W20240426 13:19:23.617424 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.988s user 0.009s sys 0.003s
I20240426 13:19:23.504515 26640 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.592353 25623 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:23.592000 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:23.596812 26683 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.600600 26658 raft_consensus.cc:979] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:23.602238 26689 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:23.602494 26601 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.561921 26654 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.604887 26483 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.573523 26677 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:23.609575 26678 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.508884 26632 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.592000 26660 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.581231 25024 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:23.618619 26551 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.721s user 0.007s sys 0.005s
I20240426 13:19:23.591774 26562 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.538924 26656 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.625537 26697 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.631927 26689 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.612097 26665 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:23.642181 26692 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240426 13:19:23.629781 26683 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.646322 26654 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.617298 26618 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.648335 26684 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:23.620239 26688 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.589927 26635 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:23.747450 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 169) took 1897 ms (client timeout 1620 ms). Trace:
I20240426 13:19:23.675455 24956 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:23.834622 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1242) took 1440 ms. Trace:
W20240426 13:19:23.831154 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.100s sys 0.104s
W20240426 13:19:23.706928 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 454) took 1335 ms (client timeout 1645 ms). Trace:
W20240426 13:19:23.895277 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.100s sys 0.113s
W20240426 13:19:23.714907 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 330879us
W20240426 13:19:23.895562 25622 rpcz_store.cc:269] 0426 13:19:22.368139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.368262 (+ 123us) service_pool.cc:225] Handling call
0426 13:19:23.703173 (+1334911us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b540033a254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.703344 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1179458432}
I20240426 13:19:23.545676 26617 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.745206 25023 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240426 13:19:23.707597 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1182) took 1782 ms (client timeout 1869 ms). Trace:
I20240426 13:19:23.897668 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:19:23.896157 24950 rpcz_store.cc:269] 0426 13:19:21.850118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.850229 (+ 111us) service_pool.cc:225] Handling call
0426 13:19:23.747238 (+1897009us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.747359 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305292544}
I20240426 13:19:23.647753 26601 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:23.595261 26676 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s user 0.002s sys 0.011s
W20240426 13:19:23.912811 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 293206us
W20240426 13:19:23.886350 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 388050us
I20240426 13:19:23.803385 26467 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.928330 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:23.868336 26507 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.748642 26664 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.852850 26589 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:23.911057 25031 rpcz_store.cc:269] 0426 13:19:21.921830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.921947 (+ 117us) service_pool.cc:225] Handling call
0426 13:19:23.704454 (+1782507us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:23.704572 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258830080}
I20240426 13:19:23.957144 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:23.980180 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:23.980984 25023 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:23.983716 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:23.742362 26625 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.962347 26683 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.744091 26699 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:23.907094 26669 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:23.867465 26354 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.898980 26551 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.899385 26660 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.899521 26688 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.654042 26677 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.906497 26697 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:24.325891 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 254170us
I20240426 13:19:24.328414 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:23.984143 25104 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:24.328783 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.335090 25031 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.335594 25025 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:24.095264 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:24.118116 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.014s sys 0.059s
I20240426 13:19:23.913795 26535 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:23.627156 26661 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:24.117928 25030 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.032s sys 0.027s
W20240426 13:19:24.337074 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s user 0.023s sys 0.067s
I20240426 13:19:24.341812 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:24.287791 26694 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.573s user 0.001s sys 0.001s
W20240426 13:19:24.345533 25030 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.045s sys 0.038s
I20240426 13:19:24.058993 25618 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:24.256058 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:23.955152 26685 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:24.317341 26611 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.760s user 0.007s sys 0.003s
I20240426 13:19:23.958227 26640 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.352892 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.197026 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.351652 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.351972 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.354254 25023 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:24.354611 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.196862 26534 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.039587 26633 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.135798 26588 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.287122 26689 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.101552 26634 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.101840 26527 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.327636 26526 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.083197 26473 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.060195 26515 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.350016 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.286170 26656 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.088786 26604 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:23.973707 26635 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:23.951225 26601 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.323817 26665 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.071262 26614 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.355357 25024 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:24.353698 25100 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.090641 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:23.916347 24948 rpcz_store.cc:276] 0426 13:19:22.394085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.394218 (+ 133us) service_pool.cc:225] Handling call
0426 13:19:22.394508 (+ 290us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:22.406897 (+ 12389us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:22.406927 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:22.681990 (+275063us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b2c00000918. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed094d2 00007fda90352f01 00007fda9031771a 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742
0426 13:19:22.682421 (+ 431us) log.cc:842] Serialized 26 byte log entry
0426 13:19:23.501299 (+818878us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:23.501349 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:23.501463 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:23.834246 (+332783us) raft_consensus.cc:1671] finished
0426 13:19:23.834273 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:23.834526 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122120,"raft.queue_time_us":30533,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":10601,"spinlock_wait_cycles":4230528,"thread_start_us":817888,"threads_started":1,"wal-append.queue_time_us":1129657}
W20240426 13:19:24.432207 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.039s sys 0.059s
W20240426 13:19:24.434579 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.521s user 0.040s sys 0.060s
I20240426 13:19:24.085641 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.347558 26685 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.339491 26562 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:23.955844 26618 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.965369 26467 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.356323 26611 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:23.984647 26710 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.388635 26679 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.408263 26508 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.364424 26633 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.352200 26617 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.059049 26589 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.386410 26625 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:24.393482 26687 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.517s user 0.002s sys 0.000s
I20240426 13:19:24.395313 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.477603 25102 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:24.449142 26654 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:24.396382 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.085809 26672 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.196337 26711 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.328943 26682 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.123303 26632 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:23.987982 26507 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:24.359589 26658 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.004s sys 0.039s
W20240426 13:19:23.987685 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 491290us
I20240426 13:19:24.437415 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.444931 26604 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.544243 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.544869 25105 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:24.545274 26604 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.550994 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 460) took 1042 ms. Trace:
I20240426 13:19:24.551548 25615 rpcz_store.cc:276] 0426 13:19:23.508535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.508862 (+ 327us) service_pool.cc:225] Handling call
0426 13:19:23.509212 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:23.956835 (+447623us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:23.956878 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:23.956933 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:23.956975 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:24.550708 (+593733us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:24.550769 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:24.550926 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1028965120}
I20240426 13:19:24.552804 24950 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:24.418061 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.435581 25103 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:23.964928 26704 raft_consensus.cc:2741] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 13:19:24.384459 26640 raft_consensus.cc:1067] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:24.439939 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:24.440994 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.438679 26413 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.442845 26697 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.381404 26588 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.438972 26441 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.443073 26688 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.349289 26535 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.355381 26661 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.385696 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.433251 26673 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.433799 26705 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.409426 26683 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.437531 26689 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.462916 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:24.466387 26714 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.466539 26713 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.439256 26526 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.437949 26503 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.410122 26660 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.436905 26656 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.411970 26626 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.474823 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:24.443086 26551 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.444756 26635 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.338732 26354 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.486112 25100 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:24.362856 26534 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.492308 26666 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.492432 26712 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.521917 25031 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:24.409525 26527 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.411672 26515 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.398365 26634 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.460707 26676 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.439801 26614 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.348496 26699 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.411501 26473 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.476483 26508 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.526566 26625 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.487260 26679 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.461380 26710 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.868213 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1188) took 1355 ms. Trace:
W20240426 13:19:24.876770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 413970us
I20240426 13:19:24.882099 24954 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:24.553686 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:24.884792 25104 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:24.886860 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.887712 25025 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:24.890779 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:24.543359 26658 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.004s sys 0.044s
W20240426 13:19:24.892375 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 349445us
W20240426 13:19:24.891259 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1236) took 2840 ms (client timeout 1504 ms). Trace:
I20240426 13:19:24.876881 25030 rpcz_store.cc:276] 0426 13:19:23.512540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.514269 (+ 1729us) service_pool.cc:225] Handling call
0426 13:19:23.562229 (+ 47960us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:24.346375 (+784146us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.346414 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:24.354178 (+ 7764us) log.cc:842] Serialized 26 byte log entry
0426 13:19:24.446839 (+ 92661us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:24.446871 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:24.446933 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:24.867905 (+420972us) raft_consensus.cc:1671] finished
0426 13:19:24.867931 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:24.868124 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17652,"raft.queue_time_us":186171,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"thread_start_us":847151,"threads_started":2,"wal-append.queue_time_us":511903}
W20240426 13:19:24.895669 24956 rpcz_store.cc:269] 0426 13:19:21.994173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.994320 (+ 147us) service_pool.cc:225] Handling call
0426 13:19:24.834596 (+2840276us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54001018d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:24.834739 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3535935744}
I20240426 13:19:24.909072 25099 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:24.872478 26467 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.865592 26601 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.762647 26562 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:24.706670 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 150) took 3011 ms (client timeout 1648 ms). Trace:
I20240426 13:19:24.912228 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:24.975595 25622 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:24.801270 26595 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.870448 26618 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.797076 26685 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.887202 26617 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:24.968780 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:24.968408 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:24.888641 26551 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:24.874161 26677 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.537111 26654 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:24.838227 26681 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.891894 26635 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.761528 26709 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.761137 26722 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.980124 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:24.889067 26626 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.981627 25025 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.012373 26709 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.014020 26626 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.014940 26681 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.909670 26666 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.017616 26681 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.837881 26718 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:24.913604 25096 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:24.972681 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.018132 26666 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:24.557507 26682 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.977821 26589 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.018237 26718 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.019212 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 200201us
I20240426 13:19:25.020450 26682 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:24.979007 25022 rpcz_store.cc:269] 0426 13:19:21.682315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.682533 (+ 218us) service_pool.cc:225] Handling call
0426 13:19:24.693424 (+3010891us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:24.693586 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4199719296}
I20240426 13:19:25.020495 26718 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.021946 26660 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:24.898157 26673 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:25.022507 26660 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.024372 26673 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.932559 26658 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.012398 26722 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.025458 26658 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.027002 26722 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.907991 26664 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.027695 26664 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.028168 26664 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.028343 26658 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.871609 26507 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.021926 25025 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:24.996688 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 534164us
I20240426 13:19:24.541548 26672 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.024516 26660 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.962985 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:24.968315 26719 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:24.894361 26688 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20240426 13:19:24.883777 26689 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.037987 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:24.890470 26441 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.887315 26526 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.043010 26710 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:24.889969 26413 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.980021 26640 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.063678 26291 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:24.891816 26683 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.895608 26656 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.046831 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.004s sys 0.028s
W20240426 13:19:25.067787 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s user 0.004s sys 0.028s
I20240426 13:19:25.031178 26601 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.029836 26611 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.064862 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.029735 26633 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.894270 26503 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.913048 26676 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.069669 26534 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.070864 26634 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:24.962272 26713 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:24.983342 26595 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.071890 26588 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.075726 26720 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.076277 26721 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.076702 26526 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.076961 26708 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:24.970667 26714 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.117942 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.009083 26551 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.046113 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 453) took 2669 ms (client timeout 2235 ms). Trace:
I20240426 13:19:25.083531 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:24.889122 26705 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.032729 26635 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:24.641642 26711 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.088277 24852 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 172) took 3087 ms. Trace:
I20240426 13:19:24.911105 26712 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.090749 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.091787 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.092361 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.095249 26672 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.096153 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.140341 24950 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:25.072525 26660 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:24.988377 26562 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.099963 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.075177 26589 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:25.115542 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.093353 26710 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20240426 13:19:25.072310 26640 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.079514 26601 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.122442 24922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.014s sys 0.036s
I20240426 13:19:25.123310 26413 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:25.134555 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 219108us
I20240426 13:19:25.137634 26595 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.143369 25023 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.147346 24954 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.130426 24852 rpcz_store.cc:276] 0426 13:19:21.994985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.995772 (+ 787us) service_pool.cc:225] Handling call
0426 13:19:25.082616 (+3086844us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240426 13:19:25.163502 25031 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:25.106659 26503 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.149641 26354 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.139941 26515 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.120118 26527 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.116954 26721 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.099577 26654 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:25.154505 24922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.014s sys 0.036s
I20240426 13:19:25.126652 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.122985 26535 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.195667 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.050360 26719 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.196349 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.208712 25622 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:25.198796 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.210743 25028 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.212767 25027 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:25.198386 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.216044 25022 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:25.217430 26716 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:25.122429 25624 rpcz_store.cc:269] 0426 13:19:22.367720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.389191 (+ 21471us) service_pool.cc:225] Handling call
0426 13:19:25.030299 (+2641108us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.036781 (+ 6482us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989924096}
I20240426 13:19:25.154925 24949 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.121078 26709 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.172528 26705 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.165565 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.161785 26656 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.166803 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.226174 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 274) took 1143 ms. Trace:
I20240426 13:19:25.169545 26712 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:25.127432 24921 connection.cc:232] RPC call timeout handler was delayed by -1.1647238229999175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
I20240426 13:19:25.171196 26635 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.154070 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.173031 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.244092 25100 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:25.221602 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.127264 26661 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.162012 26508 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.226708 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.225915 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.167997 26473 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.144205 26660 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.229012 24954 raft_consensus.cc:2354] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.150915 26711 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.206733 26614 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.217952 26676 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.237502 26683 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.229704 26689 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.154145 26708 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.164096 26679 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.148717 26625 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.276691 26719 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.278077 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.245505 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:25.246434 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.225188 26617 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.244158 25103 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:25.246831 26713 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.248556 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 213) took 1139 ms. Trace:
I20240426 13:19:25.249946 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 210) took 1165 ms. Trace:
I20240426 13:19:25.206085 26666 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.253317 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.253660 26721 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.254654 26640 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.229987 26441 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:25.238816 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 268) took 3353 ms (client timeout 2168 ms). Trace:
I20240426 13:19:25.256983 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.294749 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.221410 26716 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.231062 26699 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.261015 26562 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:25.262027 26601 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.264443 26551 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.252543 26660 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:25.265313 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 214) took 1158 ms. Trace:
I20240426 13:19:25.265666 26664 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.311187 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.268196 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.268317 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.312341 25025 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.267915 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.313097 25102 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240426 13:19:25.268831 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.269434 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.270334 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.314093 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.314472 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.314800 25102 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:25.315825 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.309844 25097 rpcz_store.cc:276] 0426 13:19:24.106922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.281393 (+174471us) service_pool.cc:225] Handling call
0426 13:19:24.344945 (+ 63552us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.913342 (+568397us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.913378 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.913430 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:24.913479 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.264941 (+351462us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b540009ced4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.265000 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.265141 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306922240}
I20240426 13:19:25.271173 25099 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.141517 26720 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.272330 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.319785 24954 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.317178 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:25.320559 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.273281 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.272334 26689 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.274876 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:25.280452 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:25.324620 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.280071 25098 rpcz_store.cc:276] 0426 13:19:24.084609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.085722 (+ 1113us) service_pool.cc:225] Handling call
0426 13:19:24.834081 (+748359us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:24.834275 (+ 194us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:25.110781 (+276506us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.110813 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:25.110856 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:25.110885 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.249653 (+138768us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54001b3854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.249711 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.249854 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":648963072}
I20240426 13:19:25.282955 25029 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.283236 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:25.283814 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 193165us
I20240426 13:19:25.326056 25622 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:25.300312 26690 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.279945 25101 rpcz_store.cc:276] 0426 13:19:24.085483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.277390 (+191907us) service_pool.cc:225] Handling call
0426 13:19:24.351784 (+ 74394us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.704458 (+352674us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.704513 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.704568 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:24.704611 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.222089 (+517478us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.222130 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.225142 (+ 3012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810539392}
I20240426 13:19:25.278098 26714 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.295884 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.306949 25023 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.308264 24920 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:25.309713 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 422847us
I20240426 13:19:25.231921 25616 rpcz_store.cc:276] 0426 13:19:24.083264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.085249 (+ 1985us) service_pool.cc:225] Handling call
0426 13:19:24.286442 (+201193us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.356537 (+ 70095us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.356578 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.360221 (+ 3643us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:24.383311 (+ 23090us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.225864 (+842553us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.225965 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.226107 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152181120}
I20240426 13:19:25.339712 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.273808 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:25.340240 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.322396 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.341262 25031 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 1.
I20240426 13:19:25.322547 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.324822 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.284050 24956 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.342598 25028 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.340281 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:25.326912 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.328262 25615 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240426 13:19:25.326967 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.344476 24953 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.344702 24948 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.285238 26635 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.329694 25103 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.337463 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:25.330309 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.309537 26625 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:25.351953 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.304589 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:25.340637 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66892.3us
I20240426 13:19:25.296902 26614 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:25.350571 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.336292 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.351395 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.354038 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.284917 26617 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.322782 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.358863 24956 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240426 13:19:25.361740 24995 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11fad756437243d2959f5998d6bf8309. This is attempt 1: this message will repeat every 5th retry.
W20240426 13:19:25.437923 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 76365.4us
I20240426 13:19:25.297068 26666 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.428362 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1182) took 1108 ms. Trace:
I20240426 13:19:25.428537 25624 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.428992 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.471889 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.468659 25099 rpcz_store.cc:276] 0426 13:19:24.320337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.387562 (+ 67225us) service_pool.cc:225] Handling call
0426 13:19:25.428178 (+1040616us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.428303 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72873856}
I20240426 13:19:25.429874 26551 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:25.430537 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 122357us
I20240426 13:19:25.476320 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.476619 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.430835 26707 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.429880 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.370882 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.431730 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:25.295570 25618 rpcz_store.cc:269] 0426 13:19:21.885828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.886511 (+ 683us) service_pool.cc:225] Handling call
0426 13:19:25.238623 (+3352112us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:25.238755 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4274581760}
I20240426 13:19:25.433087 24947 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.433635 24954 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.368579 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.360714 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.434353 26716 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.361937 26714 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.439538 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.441406 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 458) took 2172 ms. Trace:
I20240426 13:19:25.440016 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:25.331403 26690 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.491439 26690 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.489332 26714 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.454592 24996 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:25.441371 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.492758 24996 leader_election.cc:401] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20240426 13:19:25.492998 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51698.1us
I20240426 13:19:25.493355 25103 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:25.493621 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.443641 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.429448 26656 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.496378 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:25.496826 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.010s sys 0.031s
W20240426 13:19:25.497120 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.010s sys 0.031s
I20240426 13:19:25.334153 26600 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:25.448922 25105 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.442967 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.326822 26588 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.480221 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.499251 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.501145 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.484012 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.499075 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.502236 25023 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.502992 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.481256 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.503180 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.485262 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.485706 25098 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.503983 24954 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.482529 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:25.486124 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 117629us
I20240426 13:19:25.505484 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.488428 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.506105 24954 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.507809 25622 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:25.508582 25100 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.509408 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.510290 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.491874 26717 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.512378 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:25.497794 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1543) took 1049 ms. Trace:
I20240426 13:19:25.489996 25022 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.513039 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.519973 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.520980 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.521148 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.481858 26707 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:25.512509 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.524932 24948 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:25.513365 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.368672 26625 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.440156 26614 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:25.478065 25026 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.478282 26704 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20240426 13:19:25.511648 26588 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.491606 26553 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240426 13:19:25.511749 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.514380 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.548118 25031 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:25.548431 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.517901 24951 rpcz_store.cc:276] 0426 13:19:24.448686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.448806 (+ 120us) service_pool.cc:225] Handling call
0426 13:19:24.451367 (+ 2561us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.451534 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:24.736306 (+284772us) log.cc:842] Serialized 14 byte log entry
0426 13:19:25.497495 (+761189us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:25.497544 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:25.497582 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.497645 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.497748 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":517455,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":3017,"raft.queue_time_us":759,"thread_start_us":1029786,"threads_started":2,"wal-append.queue_time_us":770733}
I20240426 13:19:25.356580 26737 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:25.510875 25100 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:25.525971 24956 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:25.526391 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:25.353276 26473 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.555106 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.522541 26700 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.532778 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.535655 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.535806 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.536628 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.529433 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.498224 26413 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:25.441632 26708 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.569954 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.570750 25102 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.541054 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:25.488612 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.542443 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.352154 26689 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.544275 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.544400 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.356436 26635 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:25.545090 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 270296us
I20240426 13:19:25.545405 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.592861 24947 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.545476 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.522500 26717 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.483184 26551 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.522241 26683 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.532533 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.537165 26441 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:25.532214 26707 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.531956 26508 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.538727 26614 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:25.629607 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.564536 24949 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:25.569294 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.561362 26473 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.583760 26738 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:25.583892 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.543849 26505 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.583667 26601 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.587280 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:25.646070 25101 raft_consensus.cc:595] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.573691 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:25.489543 25617 rpcz_store.cc:276] 0426 13:19:23.269445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:23.270127 (+ 682us) service_pool.cc:225] Handling call
0426 13:19:24.389879 (+1119752us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:24.390098 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:24.968373 (+578275us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:24.968408 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:24.968455 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:24.968496 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.441087 (+472591us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:25.441153 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:25.441316 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1366731776}
I20240426 13:19:25.592785 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:25.511715 26656 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.593530 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:25.651252 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58550.4us
I20240426 13:19:25.550011 26588 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.597174 25098 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:25.556283 26737 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.653776 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.599135 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.601936 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:25.605408 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.655249 25029 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:25.607512 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.609751 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.609196 24854 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 277) took 4083 ms. Trace:
I20240426 13:19:25.585103 26718 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.618590 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:25.615568 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.568508 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:25.564205 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:25.675606 26696 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.186s
I20240426 13:19:25.595837 26413 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:25.599716 26717 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.636572 25105 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.597671 26720 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:25.640103 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.640619 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.634342 25022 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:25.642614 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119171us
I20240426 13:19:25.586109 26716 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:25.568914 26700 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:25.604619 26664 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.591008 25102 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:25.647789 26601 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:25.648232 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.629352 26508 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.636448 26473 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.631994 26441 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:25.670775 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68957.2us
I20240426 13:19:25.674203 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:25.655304 25588 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.821982 25588 leader_election.cc:401] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:25.676110 26653 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.676743 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:25.823494 25029 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.677636 26635 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.717276 26551 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.720391 26625 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.720530 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.798710 26601 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.725040 26656 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.739370 24947 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:25.866741 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:25.741025 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 176980us
I20240426 13:19:25.868289 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.868013 26413 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.869467 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:25.630499 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97637.6us
I20240426 13:19:25.754185 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.784871 25070 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:25.872253 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 105214us
I20240426 13:19:25.872709 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:25.787739 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.753979 26657 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:25.874043 26698 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:25.874459 26657 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:25.874846 26657 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.874133 26481 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.875484 26562 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:25.875538 26664 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:25.874164 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86927.3us
I20240426 13:19:25.877048 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:25.678617 26553 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.666231 26737 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:25.665401 26617 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.665699 26600 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:25.823940 25025 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:25.824692 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.684006 26666 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.783418 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:25.826614 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 171445us
I20240426 13:19:25.727514 26714 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:25.888988 26664 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:25.820649 25616 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:25.721660 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 375055us
I20240426 13:19:25.834314 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.834764 24956 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:25.814016 26508 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.959607 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:25.835453 26744 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:25.846738 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.865108 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.799108 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.880085 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:25.983440 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103449us
I20240426 13:19:25.888269 26553 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:25.948017 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 68414.6us
I20240426 13:19:25.955200 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:25.917594 26600 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:25.988883 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.833930 26653 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.995465 26441 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:25.989395 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:25.743609 26718 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:25.989948 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.993208 26600 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.994874 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.996259 26653 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:25.990125 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:25.981581 26562 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:25.997341 26553 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:25.998026 26720 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.016017 26610 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.035775 26698 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.041239 26698 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.008735 26718 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:26.047358 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 129420us
W20240426 13:19:26.048063 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59270.3us
W20240426 13:19:26.047655 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 170693us
I20240426 13:19:26.051060 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.052210 26689 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.053171 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.072082 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.072674 24954 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:26.073449 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.074013 24954 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.055732 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.057404 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.060026 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.061496 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.063751 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.009374 26562 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.068768 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:26.070191 25588 leader_election.cc:398] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.074568 24956 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:26.075270 24947 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:26.077227 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:25.996668 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.080380 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.080821 25105 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:26.068754 24949 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:26.081753 25588 leader_election.cc:401] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.084331 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.044898 26720 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.105636 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.108904 26708 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.047863 26716 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:26.114187 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44165.2us
I20240426 13:19:26.114939 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.117715 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.083482 26633 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.109884 26689 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:25.656771 24854 rpcz_store.cc:276] 0426 13:19:21.525781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.528160 (+ 2379us) service_pool.cc:225] Handling call
0426 13:19:25.609141 (+4080981us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:21.896368 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:21.987080 (+ 90712us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:21.993371 (+ 6291us) write_op.cc:427] Acquired schema lock
0426 13:19:21.999430 (+ 6059us) tablet.cc:660] Decoding operations
0426 13:19:22.059248 (+ 59818us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:22.059348 (+ 100us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:22.059389 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0426 13:19:22.059605 (+ 216us) tablet.cc:699] Row locks acquired
0426 13:19:22.059631 (+ 26us) write_op.cc:260] PREPARE: finished
0426 13:19:22.725363 (+665732us) spinlock_profiling.cc:243] Waited 902 us on lock 0x7b2c000471f8. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8ecf9ea1 00007fda8ed08b6a 00007fda8ed087f3 00007fda8ed08791 00007fda90352e02 00007fda90352bdd 00007fda93d27d10 00007fda93d27828 00007fda93d2c111 00007fda93d2c0c9 00007fda93d2c059
0426 13:19:22.725431 (+ 68us) write_op.cc:270] Start()
0426 13:19:22.725628 (+ 197us) write_op.cc:275] Timestamp: P: 1714137562060674 usec, L: 0
0426 13:19:22.725651 (+ 23us) op_driver.cc:331] REPLICATION: starting
0426 13:19:22.726244 (+ 593us) log.cc:842] Serialized 535 byte log entry
0426 13:19:25.148568 (+2422324us) op_driver.cc:444] REPLICATION: finished
0426 13:19:25.448610 (+300042us) write_op.cc:300] APPLY: starting
0426 13:19:25.448703 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:25.448816 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:25.448844 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:25.449070 (+ 226us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:25.449253 (+ 183us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0426 13:19:25.449313 (+ 60us) write_op.cc:311] APPLY: finished
0426 13:19:25.449545 (+ 232us) log.cc:842] Serialized 23 byte log entry
0426 13:19:25.449724 (+ 179us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:25.449814 (+ 90us) write_op.cc:446] Released schema lock
0426 13:19:25.449858 (+ 44us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:25.450176 (+ 318us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:25.450210 (+ 34us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1628,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":291485,"apply.run_cpu_time_us":1852,"apply.run_wall_time_us":1852,"num_ops":1,"prepare.queue_time_us":260872,"prepare.run_cpu_time_us":22973,"prepare.run_wall_time_us":1611330,"raft.queue_time_us":645897,"raft.run_cpu_time_us":4487,"raft.run_wall_time_us":213198,"replication_time_us":2415761,"spinlock_wait_cycles":2075392,"thread_start_us":1282759,"threads_started":4,"wal-append.queue_time_us":899830}]]}
I20240426 13:19:26.121541 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:26.111575 26720 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.101840 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.104585 26617 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.085600 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.107897 25069 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.133241 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.083711 26714 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20240426 13:19:26.140875 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55370.5us
I20240426 13:19:26.126361 26682 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.166890 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.112834 26747 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.118563 26744 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.157236 26595 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:26.193634 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47582.2us
I20240426 13:19:26.117141 26716 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.125579 26633 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.144141 26481 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.194578 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.203883 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.142283 26562 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.140017 26668 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.195920 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.207227 25105 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:26.197822 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.140379 26698 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.140914 26700 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.205796 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.217279 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.193563 26595 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.197407 26682 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:26.206871 26660 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.876s user 0.006s sys 0.011s
I20240426 13:19:26.188663 26747 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.230824 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.201885 26633 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.237916 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.230067 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.239035 24948 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.239759 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.231405 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.232112 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.201799 26481 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.221827 26684 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.226130 26595 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.240386 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.244424 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.216804 26664 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.226763 26700 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:26.227494 26676 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.249724 25619 raft_consensus.cc:2354] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
W20240426 13:19:26.249852 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 238926us
I20240426 13:19:26.227716 26610 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:26.255460 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 384964us
I20240426 13:19:26.251667 25589 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.250753 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.237116 26747 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.258806 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.256055 26676 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.249019 26481 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.250171 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.228421 26698 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 13:19:26.237097 26688 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:26.250958 26684 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.251114 26681 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.252646 26700 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:26.255251 25624 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.756s user 0.010s sys 0.012s
I20240426 13:19:26.271155 25624 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:26.263056 26676 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.234813 26660 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.256736 26505 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.284821 26505 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.283900 26553 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.273108 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:26.286006 26660 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.286100 26699 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.286617 26699 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.235428 26617 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 13:19:26.274245 26745 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:26.270020 26473 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.264191 26668 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.267438 26684 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:26.285094 26751 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20240426 13:19:26.266758 26657 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.292491 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:26.295758 26707 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.662s user 0.013s sys 0.003s
I20240426 13:19:26.301252 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.325368 26707 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.326197 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:26.327606 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 194451us
W20240426 13:19:26.320823 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63397.9us
I20240426 13:19:26.328841 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.272532 26700 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.331779 26481 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.332196 26682 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.332620 26682 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.332976 26682 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.323931 24949 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:26.287420 26699 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.328372 25620 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:26.327693 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.326792 26668 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.339154 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.341925 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.342739 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:26.343739 26755 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20240426 13:19:26.345906 26740 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.699s user 0.002s sys 0.000s
I20240426 13:19:26.345770 26505 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.347374 26657 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.349939 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.339365 26757 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.352499 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1539) took 1915 ms. Trace:
I20240426 13:19:26.350920 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.358345 26757 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.359725 26551 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:26.360139 26551 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.360301 26707 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.360549 26551 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.362215 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.336864 26481 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.365482 26601 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.356259 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.368193 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.352994 24955 rpcz_store.cc:276] 0426 13:19:24.437350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.437716 (+ 366us) service_pool.cc:225] Handling call
0426 13:19:24.875839 (+438123us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.043896 (+168057us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.043936 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.049254 (+ 5318us) log.cc:842] Serialized 26 byte log entry
0426 13:19:25.539822 (+490568us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:25.539873 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.540002 (+ 129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.345398 (+805396us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:26.347442 (+ 2044us) raft_consensus.cc:1671] finished
0426 13:19:26.347465 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:26.352435 (+ 4970us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22513,"raft.queue_time_us":98438,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6265,"spinlock_wait_cycles":48407552,"thread_start_us":657489,"threads_started":2,"wal-append.queue_time_us":546170}
I20240426 13:19:26.383117 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.362138 26727 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.387156 26668 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:26.387694 26505 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:26.366947 26588 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.693s user 0.015s sys 0.018s
I20240426 13:19:26.349505 26660 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:26.336864 26716 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.334026 26756 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:26.324582 26633 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.405025 24955 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.376767 26745 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.384155 26657 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.428411 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:26.396816 26754 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20240426 13:19:26.385694 26664 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:26.405395 26614 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.760s user 0.010s sys 0.005s
I20240426 13:19:26.446681 26664 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.366655 26696 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240426 13:19:26.395443 26727 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:26.451069 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110957us
I20240426 13:19:26.404691 26660 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:26.444878 26690 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.951s user 0.017s sys 0.002s
I20240426 13:19:26.371266 26595 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.441452 26716 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.426412 26745 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:26.424396 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.459149 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.440971 26657 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.468658 26595 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.468461 26657 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.466420 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.446185 26441 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240426 13:19:26.404542 26588 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.399943 26508 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.455963 26505 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.451869 26756 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.454042 26664 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.475548 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.478655 26473 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.460928 26660 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:26.487875 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:26.489888 26737 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.573s user 0.025s sys 0.024s
I20240426 13:19:26.490653 25030 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.490834 26553 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.500528 26633 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.500605 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.474879 26716 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.477247 26635 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.507980 26635 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.470908 26745 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.508821 26635 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.498667 26656 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:26.499198 26653 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.519379 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:26.519980 25099 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.491180 26625 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.523401 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.462966 26690 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.506651 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1244) took 1230 ms. Trace:
I20240426 13:19:26.507485 26601 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.517652 25022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51378
I20240426 13:19:26.457710 26614 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.524560 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:26.494083 26505 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.526605 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.533821 25023 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.529359 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.537531 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.536717 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.502993 26551 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.539670 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.540656 25096 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:26.526216 24953 rpcz_store.cc:276] 0426 13:19:25.280873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.353214 (+ 72341us) service_pool.cc:225] Handling call
0426 13:19:25.467865 (+114651us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.721372 (+253507us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.721430 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.740751 (+ 19321us) log.cc:842] Serialized 26 byte log entry
0426 13:19:26.050364 (+309613us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:26.050632 (+ 268us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:26.502919 (+452287us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:26.502968 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.503014 (+ 46us) raft_consensus.cc:1671] finished
0426 13:19:26.503028 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:26.506582 (+ 3554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582402304,"thread_start_us":262571,"threads_started":1,"wal-append.queue_time_us":381491}
I20240426 13:19:26.554261 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.540310 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.530179 26679 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.504616 26737 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.527984 26660 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:26.569403 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44901.1us
I20240426 13:19:26.574963 26772 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:26.588397 26600 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.562s user 0.014s sys 0.004s
I20240426 13:19:26.588865 26600 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.524211 26775 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:26.536247 26588 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.579924 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.582624 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.583364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.490686 26668 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.593704 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.594897 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.589777 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.595271 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.596961 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.598075 25101 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.572824 26679 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:26.575348 26653 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.613898 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.589762 26716 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.589829 26772 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.616189 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.616458 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.621971 25097 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.614692 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.607698 26551 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.618921 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.626824 26676 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.631297 25101 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:26.631947 26508 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.632294 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.618665 25624 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.619100 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:26.637051 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 175472us
I20240426 13:19:26.639570 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.609162 26679 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.619100 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.645592 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.648633 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:26.617131 25026 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.066s user 0.009s sys 0.034s
W20240426 13:19:26.621385 26718 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s user 0.011s sys 0.007s
W20240426 13:19:26.617609 26717 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.824s user 0.015s sys 0.005s
I20240426 13:19:26.660624 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.662004 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.664429 26666 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:26.664938 26761 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20240426 13:19:26.664952 26666 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.665563 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.666592 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.667065 25104 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.667201 26786 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:26.620303 26614 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.668699 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.633167 26772 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.669166 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.676461 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.677006 24955 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.619433 26660 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.640369 26653 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:26.693655 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.671623 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.673274 25029 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.668197 26786 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.670256 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.696599 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.683707 25619 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240426 13:19:26.575438 26756 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.676136 26688 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20240426 13:19:26.667176 26666 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.704762 26700 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.706646 26700 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.708559 25022 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:26.709124 25022 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.708388 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.707763 25097 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:26.701646 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.711289 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.712339 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.715626 24955 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:26.719830 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 50740.5us
I20240426 13:19:26.720222 26714 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.720378 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.720578 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.598397 26775 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.721458 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.722791 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.738047 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.738613 25028 raft_consensus.cc:2354] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.651412 25026 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:26.722748 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.723664 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.742372 26291 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.724900 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.725661 26777 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.744706 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.725864 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:26.726240 26780 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20240426 13:19:26.743985 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.728981 26505 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.729116 26747 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.643328 26773 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20240426 13:19:26.753159 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:26.735343 26676 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.721951 24995 leader_election.cc:398] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
W20240426 13:19:26.699193 26778 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20240426 13:19:26.725906 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.741751 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.747083 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.718274 26653 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.709635 26786 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:26.717864 26657 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.745965 26656 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.718170 26551 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.755977 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.752744 25104 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:26.669432 26718 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.758373 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.766206 26505 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.669615 26717 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.773272 25615 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 13:19:26.774349 25027 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.724604 26625 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:26.752595 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 169) took 1480 ms. Trace:
I20240426 13:19:26.777776 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.751267 26690 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.750967 26553 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.782495 25028 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.781622 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.770928 26721 raft_consensus.cc:979] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:26.828248 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.722014 26635 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20240426 13:19:26.829717 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.781720 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:26.762566 25619 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:26.833101 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.733974 26684 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:26.875468 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 92027.8us
I20240426 13:19:26.758782 26785 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:26.771054 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:26.768671 26737 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:26.843643 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65935us
W20240426 13:19:26.847827 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 584212us
I20240426 13:19:26.862435 25617 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.884606 26745 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.886231 25616 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:26.851459 26505 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.776934 26786 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.871146 26653 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.888636 26786 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:26.864856 26772 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.866739 26718 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.868288 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:26.859297 26720 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.714s user 0.014s sys 0.014s
I20240426 13:19:26.889446 26704 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.893491 26720 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.895371 25027 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20240426 13:19:26.869076 26682 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.536s user 0.015s sys 0.004s
I20240426 13:19:26.827827 26789 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:26.896282 26789 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.897706 25098 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:26.898278 26789 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.898763 26682 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.899259 26772 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.900597 26682 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.867738 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.903328 25101 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.873755 24952 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:26.842769 25026 rpcz_store.cc:276] 0426 13:19:25.266123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.285310 (+ 19187us) service_pool.cc:225] Handling call
0426 13:19:26.746220 (+1460910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:26.872345 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.876186 24995 leader_election.cc:398] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.909276 24995 leader_election.cc:401] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.909447 25028 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:26.876129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94507.7us
I20240426 13:19:26.878721 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.878909 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.879587 26774 raft_consensus.cc:1067] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:26.912197 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 440) took 1085 ms. Trace:
I20240426 13:19:26.912580 25620 rpcz_store.cc:276] 0426 13:19:25.826883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.827422 (+ 539us) service_pool.cc:225] Handling call
0426 13:19:26.912008 (+1084586us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:26.912142 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":794898688}
I20240426 13:19:26.913640 26721 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:26.913990 26721 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:26.914739 26721 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.915791 26721 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:26.926823 26745 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.882256 26684 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.885078 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.929899 25104 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:26.865331 26664 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.931973 26664 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.840437 26600 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.932832 25025 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:26.905534 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.906944 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.934036 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:26.899209 24920 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.934926 24920 leader_election.cc:401] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.935382 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1268) took 1623 ms. Trace:
I20240426 13:19:26.937559 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:26.938402 26600 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.938889 26600 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:26.908630 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1611) took 1231 ms. Trace:
I20240426 13:19:26.941427 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.941972 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.910115 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:26.942816 24921 leader_election.cc:398] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:26.890352 25618 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:26.917934 25026 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:26.946610 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.918112 26717 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.918336 26756 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:26.918601 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:26.880309 26747 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:26.948026 26600 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.949343 26707 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:26.904721 26785 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:26.921855 26798 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20240426 13:19:26.922617 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46584.2us
W20240426 13:19:26.947917 26750 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.834s user 0.000s sys 0.005s
I20240426 13:19:26.954080 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.955627 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.957764 26747 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:26.959321 26785 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:26.962997 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.923624 26654 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:26.924302 24851 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 287) took 4426 ms. Trace:
I20240426 13:19:26.895375 26720 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:26.925217 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:26.927960 25022 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:26.881950 26771 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:26.943828 24951 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20240426 13:19:26.940276 25023 rpcz_store.cc:276] 0426 13:19:25.676750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.677803 (+ 1053us) service_pool.cc:225] Handling call
0426 13:19:25.678063 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:26.536006 (+857943us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:26.908448 (+372442us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54002fb054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:26.908585 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1967849216}
I20240426 13:19:26.935719 25096 rpcz_store.cc:276] 0426 13:19:25.312117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.314271 (+ 2154us) service_pool.cc:225] Handling call
0426 13:19:26.935224 (+1620953us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540001ccd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:26.935331 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2818563456}
I20240426 13:19:26.971083 25620 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:26.972098 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:26.945165 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 66351.4us
I20240426 13:19:26.945533 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.916960 26775 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:26.950627 25070 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:26.951870 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.974438 25070 leader_election.cc:401] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:26.950155 26718 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.969575 25030 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 4. Ops: []
I20240426 13:19:26.949901 26707 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:26.966979 24851 rpcz_store.cc:276] 0426 13:19:22.497452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:22.497630 (+ 178us) service_pool.cc:225] Handling call
0426 13:19:26.924197 (+4426567us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20240426 13:19:26.953580 26691 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:26.970497 24853 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 2 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:26.983017 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:26.973711 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.967876 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:26.975098 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54243.1us
I20240426 13:19:26.976213 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:26.984562 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.964530 26747 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:26.988510 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:26.989123 26291 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:26.993500 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.995514 26291 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:26.992869 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:26.995908 26291 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:26.999631 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.000350 25023 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:26.997675 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.001574 25103 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:26.982347 26720 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:26.962565 26774 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:27.002182 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.003230 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.005225 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.978183 26771 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:27.003532 26291 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.002604 25620 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:27.006659 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.991384 26553 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.015291 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.013723 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:26.986305 26691 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.013305 25096 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.012077 26291 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.991554 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:27.017999 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.019678 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.020983 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.025890 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.027473 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.008983 25616 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:26.984839 26679 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.030931 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.008306 26689 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.987532 26775 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 13:19:26.991875 26713 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240426 13:19:27.031724 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.039484 24952 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.011451 26507 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.041323 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:27.018213 26795 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:26.996577 26617 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20240426 13:19:27.022188 26691 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:26.995218 26664 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240426 13:19:27.024263 26413 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.048169 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:27.029644 26553 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.007064 26747 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20240426 13:19:27.041168 26588 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.031425 26551 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20240426 13:19:27.055536 26727 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.056747 26799 ts_tablet_manager.cc:902] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.031955 26690 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.037962 26696 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.042615 26772 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.011581 26610 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240426 13:19:27.019212 26707 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.071118 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1602) took 1513 ms. Trace:
W20240426 13:19:27.080987 26794 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20240426 13:19:27.089270 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1294) took 1099 ms. Trace:
I20240426 13:19:27.111176 25098 rpcz_store.cc:276] 0426 13:19:25.989565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.989701 (+ 136us) service_pool.cc:225] Handling call
0426 13:19:27.089073 (+1099372us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:27.089212 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2063662464}
I20240426 13:19:27.089517 25097 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.100000 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.100366 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.099452 26413 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:27.051801 26771 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.114025 26413 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.100072 26601 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.090480 26610 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.094401 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:27.124603 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 92968.4us
I20240426 13:19:27.113301 25103 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:27.126931 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.127010 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.127702 25103 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:27.129084 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.129402 25096 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.129774 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.100767 26720 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.130373 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:27.132289 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:27.124564 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 95246.2us
I20240426 13:19:27.133162 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.134212 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:27.124536 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98784.2us
I20240426 13:19:27.135363 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.102844 25031 rpcz_store.cc:276] 0426 13:19:25.557361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:25.557523 (+ 162us) service_pool.cc:225] Handling call
0426 13:19:25.585195 (+ 27672us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:25.585513 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:25.585542 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:25.595605 (+ 10063us) log.cc:842] Serialized 26 byte log entry
0426 13:19:25.765641 (+170036us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:25.765693 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:25.765777 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:26.947222 (+1181445us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:27.044378 (+ 97156us) raft_consensus.cc:1671] finished
0426 13:19:27.044399 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:27.071037 (+ 26638us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1804,"raft.queue_time_us":305,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11330,"spinlock_wait_cycles":7275264,"thread_start_us":123873,"threads_started":1,"wal-append.queue_time_us":493965}
I20240426 13:19:27.089125 26679 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.137516 25022 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:27.137209 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.140427 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.143797 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.145342 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.145473 24996 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.147221 26664 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.148794 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.120525 26601 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.149633 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.148316 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.148277 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.123844 26610 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.126932 26413 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:27.161407 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144073us
I20240426 13:19:27.161736 26413 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.101795 26756 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240426 13:19:27.162050 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.163404 26777 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.162963 26748 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:27.136693 26720 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.138607 26713 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.158872 26610 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.159708 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.160317 26664 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.175496 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:27.154235 26551 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.158017 25030 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.137820 26727 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.165205 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.135833 26747 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:27.171209 26699 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.172780 25620 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.148577 26696 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.175134 26291 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.160097 26601 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.154429 26772 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:27.158357 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.157546 26505 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.178052 25100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:27.199594 25100 raft_consensus.cc:595] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.158192 26679 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.182646 26481 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:27.157867 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.186870 25623 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.201171 26481 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:27.187741 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.201586 26481 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.202930 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.204357 26736 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.204367 26795 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.204799 26716 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:27.188900 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.192338 26551 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240426 13:19:27.194882 26800 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.146s user 0.005s sys 0.000s
I20240426 13:19:27.210723 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:27.212594 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75606.1us
W20240426 13:19:27.213670 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 55436.1us
I20240426 13:19:27.214334 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.216346 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.217491 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.199183 26630 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.223006 26630 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.199422 25618 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.225315 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.226348 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.227288 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.233314 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.234560 26291 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.195312 26684 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.239369 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.209638 26481 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.265398 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107600us
I20240426 13:19:27.281791 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.282930 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.239369 26789 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.269179 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.200537 26772 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.198406 26747 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.283921 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.284570 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.289659 25100 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:27.290935 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 80299.4us
I20240426 13:19:27.263947 25104 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:27.240070 26505 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.271262 26601 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.223817 26630 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.342818 26630 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.342001 26791 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.235138 26696 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.235173 26610 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:27.225011 26551 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.198333 26777 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20240426 13:19:27.360251 26748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:47158 (request call id 6) took 5583 ms. Trace:
I20240426 13:19:27.271075 26681 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:27.296207 25030 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.385119 25100 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:27.344967 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 53051.4us
I20240426 13:19:27.224617 26699 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.266546 26625 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.270620 26688 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.272063 26756 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.290863 26776 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240426 13:19:27.295473 26751 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240426 13:19:27.295390 26737 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.292801 26775 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.292448 26772 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:27.360208 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:27.347859 26635 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.235002 26720 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.365429 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96330.5us
I20240426 13:19:27.365686 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.367885 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.368866 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.370270 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.380635 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:27.344439 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60597.9us
I20240426 13:19:27.353899 26698 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240426 13:19:27.512666 26698 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:27.513051 26698 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.513908 26698 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.515208 26775 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:27.387730 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.274109 26666 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.414654 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.415025 26774 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:27.517498 25028 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:27.385861 26777 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:27.422582 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:27.518724 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96259.7us
I20240426 13:19:27.521642 26688 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:27.522161 26688 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.474511 26776 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:27.522967 26776 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:27.524412 25023 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:27.526628 25028 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.526664 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.468963 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.438124 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.528226 24954 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.528615 24954 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:27.532747 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:27.471549 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20240426 13:19:27.518115 26777 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.542424 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:27.523016 26688 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.529503 24954 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.564841 24954 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:27.548863 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.548846 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.435366 26738 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:27.551821 25030 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 5
I20240426 13:19:27.426860 26689 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:27.570571 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:27.570574 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.527654 25031 raft_consensus.cc:2362] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4. Re-sending same reply.
I20240426 13:19:27.425544 26700 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:27.544474 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.543998 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.573915 25030 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20240426 13:19:27.692850 25030 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:27.511821 26791 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.578054 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:27.671315 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 170616us
W20240426 13:19:27.575850 26761 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.144s user 0.003s sys 0.000s
I20240426 13:19:27.576716 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.723312 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:27.408314 26747 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.709465 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 136771us
I20240426 13:19:27.571345 26738 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:27.721956 24947 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:27.724182 24948 raft_consensus.cc:2354] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:27.710424 24951 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.736346 24947 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:27.736969 24947 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.593417 26481 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:27.507090 26718 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.521s user 0.013s sys 0.000s
W20240426 13:19:27.716975 26664 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.537s user 0.009s sys 0.001s
I20240426 13:19:27.743659 24954 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:27.745012 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.713291 26600 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.567833 26691 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.734330 26711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.035s sys 0.044s
I20240426 13:19:27.789651 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.713824 26700 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.763399 26718 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:27.398900 26748 rpcz_store.cc:276] 0426 13:19:21.771338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:21.771510 (+ 172us) service_pool.cc:225] Handling call
0426 13:19:21.891393 (+119883us) log.cc:842] Serialized 492 byte log entry
0426 13:19:27.354744 (+5463351us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116470,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":5469,"thread_start_us":1607616,"threads_started":2,"wal-append.queue_time_us":396709,"wal-append.run_cpu_time_us":1887,"wal-append.run_wall_time_us":1001913}
I20240426 13:19:27.756006 25624 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.775892 26664 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:27.787211 26786 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.898s user 0.014s sys 0.001s
W20240426 13:19:27.777338 26711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.037s sys 0.046s
W20240426 13:19:27.810618 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65731.9us
W20240426 13:19:27.809566 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 276831us
I20240426 13:19:27.831161 24956 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:27.814862 26717 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:27.826289 26610 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:27.845350 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.792172 26738 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.849751 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.839658 26775 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:27.857255 24901 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:27.866320 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.866878 25102 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:27.854926 26635 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.855432 26717 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:27.870186 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 478) took 1185 ms. Trace:
I20240426 13:19:27.876121 25623 rpcz_store.cc:276] 0426 13:19:26.684408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.684564 (+ 156us) service_pool.cc:225] Handling call
0426 13:19:26.917279 (+232715us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:27.201566 (+284287us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:27.427733 (+226167us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400491954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:27.870002 (+442269us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400491950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:27.870123 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1154048768}
I20240426 13:19:27.882295 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.847551 26610 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.892776 24947 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.894647 25102 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:27.898924 25105 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:27.899641 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:27.886451 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:27.899653 24947 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:27.908164 26748 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:27.908756 26748 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:27.845878 26816 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240426 13:19:27.878283 26691 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.862895 26775 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:27.886363 26823 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:27.858640 26773 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 3 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:27.924707 25030 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 13:19:27.930327 25030 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.874847 26684 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20240426 13:19:27.933701 25031 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:27.910526 26717 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:27.933758 25028 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.864354 26817 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20240426 13:19:27.935899 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:27.941560 25025 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:27.941807 25102 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:27.942533 25030 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.944140 25102 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:27.895545 26635 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240426 13:19:27.909694 26748 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:27.922699 26700 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:27.953625 25105 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:27.916476 26562 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20240426 13:19:27.919087 26791 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240426 13:19:27.934514 26823 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:27.959360 26819 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20240426 13:19:27.965623 25102 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:27.939074 26625 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240426 13:19:27.968757 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:27.972162 25102 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:27.978372 25105 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:27.973559 26823 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.965449 26600 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:27.965514 26684 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:27.990244 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72726.4us
I20240426 13:19:27.990424 25102 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:27.982571 26817 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.012382 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:28.013891 26822 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 13:19:27.979832 26617 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240426 13:19:28.011260 26776 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:28.023675 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 474904us
I20240426 13:19:27.998920 26816 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:28.021471 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.026067 25028 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:28.026592 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.001416 26610 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:27.999650 26817 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.036474 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.037112 25025 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:28.036758 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:27.989907 26684 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.039031 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.040858 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
W20240426 13:19:28.042167 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 120716us
I20240426 13:19:28.042955 26833 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:27.987818 26600 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.040349 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.024724 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.030865 26617 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240426 13:19:28.028934 26816 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.050019 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.029126 26700 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:28.050647 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.085603 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.085974 24947 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.088006 26707 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.041648 26610 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.029788 26799 tablet_copy_client.cc:308] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Beginning tablet copy session from remote peer at address 127.23.61.2:43671
I20240426 13:19:28.039278 26775 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.097126 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:28.098299 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:28.095999 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45430.7us
I20240426 13:19:28.097739 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.102191 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.084606 26625 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.168443 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.171217 25029 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:28.095515 26700 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:28.176740 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70315.2us
I20240426 13:19:28.093938 26691 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:28.103943 26750 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20240426 13:19:28.185492 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83395.9us
I20240426 13:19:28.113533 26635 consensus_queue.cc:1225] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:19:28.095299 26825 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:28.094537 26828 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:28.101994 26757 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.108338 26839 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.175787 26744 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:28.186480 26813 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20240426 13:19:28.095891 26774 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20240426 13:19:28.185364 26691 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.185176 26503 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.196009 26825 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.177326 26708 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20240426 13:19:28.206341 26837 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20240426 13:19:28.210578 25030 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.212287 26681 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.198221 26551 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:28.177968 26816 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.197077 26833 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240426 13:19:28.222029 26630 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.879s user 0.005s sys 0.007s
I20240426 13:19:28.193276 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.223245 24951 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.227066 26691 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.215747 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 201) took 1244 ms. Trace:
I20240426 13:19:28.255622 24956 rpcz_store.cc:276] 0426 13:19:26.971607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.971900 (+ 293us) service_pool.cc:225] Handling call
0426 13:19:28.215540 (+1243640us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:28.215674 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1897286016}
I20240426 13:19:28.227820 26503 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.243211 24951 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.223331 26684 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.203105 26828 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.236465 25097 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.098834 26817 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.231812 26757 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:28.232575 26825 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.260798 25100 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.261282 26503 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.266435 25022 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 13:19:28.269129 26816 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.271168 25101 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:28.246717 26747 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240426 13:19:28.277052 26829 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:28.237722 26681 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.258745 24951 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.277395 26698 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.279495 26840 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20240426 13:19:28.291021 26757 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.294921 24949 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240426 13:19:28.277853 26776 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.278308 26849 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20240426 13:19:28.305774 24949 pending_rounds.cc:76] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:28.322731 24949 pending_rounds.cc:97] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 13:19:28.323087 24949 raft_consensus.cc:2866] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:28.300881 24951 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.301400 26508 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20240426 13:19:28.302309 26707 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:28.297427 26829 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.331532 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:28.333215 26832 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20240426 13:19:28.334407 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.335484 24956 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:28.324596 26772 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.828s user 0.012s sys 0.000s
I20240426 13:19:28.339538 26744 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:28.330091 26600 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.354334 24951 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:28.356316 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 181748us
I20240426 13:19:28.301662 26691 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.353170 24953 raft_consensus.cc:2932] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:28.359078 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.361112 26700 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.362920 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.354962 26839 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:28.359917 26562 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240426 13:19:28.363663 26848 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:28.361176 26600 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:28.324697 26630 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240426 13:19:28.376057 26600 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:28.378198 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 272044us
I20240426 13:19:28.373971 26610 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.374541 26839 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.331488 26707 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.293646 26828 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.349584 26772 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.367312 26718 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.381747 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.380009 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.363324 26507 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.410108 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.364005 26789 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.367131 26857 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
W20240426 13:19:28.418138 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 231705us
I20240426 13:19:28.422040 25099 raft_consensus.cc:1261] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:28.423556 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.425081 25041 tablet_copy_service.cc:140] P 46106ff992984d33bb4a4d8f30162639: Received BeginTabletCopySession request for tablet 11fad756437243d2959f5998d6bf8309 from peer 1590f80bdade41c6a755b8501c55fe16 ({username='slave'} at 127.0.0.1:47174)
I20240426 13:19:28.376899 26600 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.428022 26855 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.436808 26855 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.437357 26601 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:28.438755 26855 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.439271 26855 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:28.440047 25098 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.428249 25041 tablet_copy_service.cc:161] P 46106ff992984d33bb4a4d8f30162639: Beginning new tablet copy session on tablet 11fad756437243d2959f5998d6bf8309 from peer 1590f80bdade41c6a755b8501c55fe16 at {username='slave'} at 127.0.0.1:47174: session id = 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309
I20240426 13:19:28.443656 26829 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.446203 26617 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.446730 26617 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.410969 26718 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.416698 26774 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:28.417624 26789 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:28.417137 26507 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.446360 26839 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.453354 26851 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:28.477260 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119618us
W20240426 13:19:28.484874 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46310.4us
I20240426 13:19:28.485404 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.473840 26696 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.502476 26507 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.508165 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:28.517494 25104 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.517971 25104 raft_consensus.cc:603] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:28.457309 26681 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.531494 26707 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.532018 25099 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.483891 26851 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.533740 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:28.535461 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:28.536084 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:28.418957 26744 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.548161 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.515302 26617 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.553895 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.560422 25024 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.498279 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.556377 26773 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.558257 26858 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.559082 26828 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.550954 26691 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:28.581933 26863 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240426 13:19:28.566113 26825 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.585356 25028 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.571143 25104 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240426 13:19:28.520545 26789 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.526211 26696 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.498673 26718 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.579267 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.627405 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1598) took 1086 ms. Trace:
I20240426 13:19:28.590688 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 186) took 1223 ms. Trace:
I20240426 13:19:28.619402 25099 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:28.638283 25022 rpcz_store.cc:276] 0426 13:19:27.367364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.367550 (+ 186us) service_pool.cc:225] Handling call
0426 13:19:28.590502 (+1222952us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:28.590620 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1738735360}
I20240426 13:19:28.590083 25618 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.620921 26684 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:28.615820 26826 raft_consensus.cc:979] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:28.654134 26684 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:28.592777 26720 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.636816 24949 rpcz_store.cc:276] 0426 13:19:27.541004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.732154 (+191150us) service_pool.cc:225] Handling call
0426 13:19:27.732541 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.323469 (+590928us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:28.627112 (+303643us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:28.627331 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1292505856}
I20240426 13:19:28.603412 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.658916 25028 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:28.654366 25100 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.570719 26773 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.655370 26684 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.665951 26684 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.589905 26825 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.668489 26765 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240426 13:19:28.577163 26772 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.678814 26688 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:28.681293 26689 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.680680 26777 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.598966 26817 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.685633 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.686998 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.687606 24952 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:28.688613 24952 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:28.689327 26507 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.689679 24948 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:28.692713 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.692869 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.693332 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.693806 25618 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240426 13:19:28.693925 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:28.694006 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:28.661868 26630 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 13:19:28.694607 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:28.695844 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.698103 25101 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.591724 25041 tablet_copy_source_session.cc:205] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Tablet Copy: opened 0 blocks and 1 log segments
I20240426 13:19:28.681763 26689 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.704807 26773 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.699995 25028 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:28.701478 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.712213 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:28.620069 26681 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:28.715531 26773 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.717049 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1332) took 1329 ms. Trace:
I20240426 13:19:28.621186 26851 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:28.713670 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:28.719713 25098 rpcz_store.cc:276] 0426 13:19:27.387561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.387701 (+ 140us) service_pool.cc:225] Handling call
0426 13:19:27.388004 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.440457 (+1052453us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:28.716779 (+276322us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000ea454. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:28.716982 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2393130240}
W20240426 13:19:28.725032 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 265330us
W20240426 13:19:28.728338 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 315884us
W20240426 13:19:28.739559 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 134180us
I20240426 13:19:28.714174 26718 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:28.743166 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.746526 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.750770 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.752280 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.754801 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:28.758971 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:28.759306 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.761940 26850 raft_consensus.cc:1067] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:28.763739 25022 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:28.767426 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.768260 25028 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.729141 26617 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:28.770900 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.773198 25104 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:28.737624 26481 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:28.743069 26738 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.782626 25025 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20240426 13:19:28.779227 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53282.3us
I20240426 13:19:28.784745 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.785804 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.786993 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:28.790421 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.706s (SENT) [suppressed 8 similar messages]
I20240426 13:19:28.792507 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:28.794377 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50580.3us
I20240426 13:19:28.796468 26851 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:28.797513 26851 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:28.755681 26711 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:28.801301 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.802186 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.775914 26664 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:28.805109 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:28.805843 25024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:28.820657 25024 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:28.821043 25024 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.774499 26617 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:28.808321 26721 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:28.808566 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:28.810120 26826 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.810717 26772 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:28.812525 25097 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:28.778280 26481 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:28.801636 26653 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:28.743461 26816 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240426 13:19:28.777997 26756 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:28.807801 26789 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.598249 26858 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.752157 26666 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240426 13:19:28.756866 26700 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240426 13:19:28.781417 26505 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20240426 13:19:28.831634 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:28.769765 26551 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240426 13:19:28.823906 26791 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20240426 13:19:28.912578 26625 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.823696 26717 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Stepping down as leader of term 2
W20240426 13:19:28.912169 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 499599us
I20240426 13:19:28.833110 26617 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:28.915742 26772 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.807804 26441 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20240426 13:19:28.795095 26757 consensus_queue.cc:1035] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20240426 13:19:28.857023 25105 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:28.833777 26721 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.890267 26481 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.930373 26505 consensus_queue.cc:1225] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20240426 13:19:28.937281 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 104241us
W20240426 13:19:28.888932 24953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.019s sys 0.065s
I20240426 13:19:28.823289 26711 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.910146 26775 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.928921 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:28.914952 26698 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.962495 26698 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:28.806106 26840 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20240426 13:19:28.912251 26826 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:28.967133 26698 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.929769 26717 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.934000 26861 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:28.922446 26756 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:28.972433 26756 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.975267 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:28.976442 25028 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:28.943909 24953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.020s sys 0.065s
I20240426 13:19:28.930305 26696 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.961016 26865 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:28.946735 25024 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:28.961980 26711 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:28.987113 25023 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:28.987540 25069 leader_election.cc:398] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:28.992655 25024 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.017031 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.022459 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:29.023142 25105 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:29.023459 25105 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:29.024187 25105 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.025233 25105 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.026516 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.030876 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:28.938908 26707 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:28.984642 26708 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:29.035800 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:28.950335 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:28.934700 26625 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.035317 26503 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:29.043109 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.004s sys 0.016s
W20240426 13:19:29.043746 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s user 0.004s sys 0.016s
W20240426 13:19:29.044252 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 524291us
I20240426 13:19:29.046782 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.055436 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:29.034719 26776 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:29.041599 26707 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.044708 26507 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.040131 26551 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.017269 26696 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:28.928743 26852 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20240426 13:19:29.037410 26505 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.060593 24949 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.068384 24950 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:28.996384 26775 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.049319 26858 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.065155 26707 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:29.073781 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.075184 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.077275 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:29.084836 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.051191 26441 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.121927 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.062222 26666 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.025568 26721 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.125002 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 370322us
I20240426 13:19:29.042066 26711 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:19:29.087103 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.070693 26698 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.134429 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:29.079154 26600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.017s sys 0.005s
I20240426 13:19:29.120265 26505 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.080008 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.148499 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.138211 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.166484 24952 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.165395 25623 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:29.060765 26826 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.144656 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:29.085805 26610 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:29.171754 24950 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.130667 26617 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.081571 26551 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.079428 26507 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.025751 26865 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.148308 26721 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:29.165387 26600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.017s sys 0.009s
I20240426 13:19:29.062491 26772 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.192039 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.190260 25105 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:29.175344 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.056s sys 0.157s
W20240426 13:19:29.184976 26684 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.518s user 0.011s sys 0.000s
W20240426 13:19:29.191967 26744 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.011s sys 0.023s
I20240426 13:19:29.203680 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:29.202383 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.056s sys 0.158s
I20240426 13:19:29.157249 26858 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20240426 13:19:29.216637 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:29.219023 26689 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.513s user 0.009s sys 0.001s
W20240426 13:19:29.209252 26744 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.012s sys 0.023s
I20240426 13:19:29.175920 26698 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.216282 26817 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.532s user 0.009s sys 0.000s
I20240426 13:19:29.125838 26441 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.189045 26610 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.210891 26630 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.140048 26666 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.215579 26748 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.236119 26505 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.236409 25099 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:29.236749 26625 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.219931 26684 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:29.219944 26681 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.221810 26562 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.238876 26869 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.226758 26711 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.210144 26503 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20240426 13:19:29.219471 26689 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:29.240484 26610 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:29.259032 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 468514us
I20240426 13:19:29.241150 26551 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.238202 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:29.251758 24951 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.020s sys 0.021s
I20240426 13:19:29.252360 26562 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.290573 24951 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.020s sys 0.022s
I20240426 13:19:29.252506 26684 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:29.292677 25104 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:29.238127 26630 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.302104 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 347225us
I20240426 13:19:29.240430 26748 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.316691 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.319518 25099 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.319913 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.251176 26681 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.326709 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.328521 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.322304 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.323791 25101 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:29.329874 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:29.292155 26775 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:29.325644 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3fe8f3015c01480fb67b39d3f5a947c7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.333508 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3fe8f3015c01480fb67b39d3f5a947c7, TS 30c2bb0adfa54bbfa3cb39773d4043c8 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.325840 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.331596 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.332504 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.319301 26505 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.346668 26481 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.348225 26551 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.348981 26630 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.348836 26551 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.350536 26748 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.315542 26617 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.350975 26861 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.328572 25617 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:29.351284 26684 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.355515 24950 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:29.361541 25024 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:29.362030 26698 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.350920 26551 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.348992 26689 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.348256 26507 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:29.362819 26562 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.364176 26441 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:29.329274 26720 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.673s user 0.010s sys 0.000s
I20240426 13:19:29.332100 26600 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.263746 26869 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.395224 26775 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.395790 26775 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.396309 26869 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.397303 26699 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:29.397727 26699 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:29.394134 25099 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:29.329262 26817 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.401293 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.402385 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.402508 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.404003 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.404459 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.413443 25101 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:29.416021 25031 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:29.422339 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:29.388949 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.410283 26825 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.412760 26698 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:29.430738 26718 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.678s user 0.010s sys 0.000s
I20240426 13:19:29.427124 26817 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20240426 13:19:29.470247 26873 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20240426 13:19:29.401651 26600 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.461410 26707 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.473306 26600 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.474030 26707 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.475479 24948 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:29.461489 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:29.452023 25025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.007s sys 0.016s
W20240426 13:19:29.477124 25025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.008s sys 0.016s
I20240426 13:19:29.490377 24955 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:19:29.490882 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 265026us
I20240426 13:19:29.493811 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:29.466465 26882 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:29.503906 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.510669 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:29.511760 25024 raft_consensus.cc:2396] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240426 13:19:29.495637 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.516012 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:29.537344 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2d6f5c5283464089ad1bc18c13b427cc, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.537866 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2d6f5c5283464089ad1bc18c13b427cc, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.557132 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.506341 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:29.563280 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:29.563979 25101 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 4
W20240426 13:19:29.564986 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52415.5us
I20240426 13:19:29.495484 26441 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:29.565516 24953 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.032s sys 0.049s
W20240426 13:19:29.573148 24953 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.032s sys 0.049s
I20240426 13:19:29.499298 26505 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:29.578282 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1304) took 2014 ms. Trace:
I20240426 13:19:29.499673 26481 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.501545 26413 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:29.468506 26698 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.560006 26789 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.622s user 0.009s sys 0.001s
I20240426 13:19:29.594564 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.465412 26825 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.604673 25099 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.594686 24953 rpcz_store.cc:276] 0426 13:19:27.563820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:27.727674 (+163854us) service_pool.cc:225] Handling call
0426 13:19:28.352716 (+625042us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:28.352908 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.353079 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:28.384310 (+ 31231us) log.cc:842] Serialized 14 byte log entry
0426 13:19:29.577877 (+1193567us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:29.577937 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:29.577993 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.578070 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:29.578205 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18047,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":21683,"spinlock_wait_cycles":5111680,"thread_start_us":1193178,"threads_started":2,"wal-append.queue_time_us":455874}
I20240426 13:19:29.567898 26707 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.601119 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240426 13:19:29.537504 26654 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:29.601583 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 390531us
I20240426 13:19:29.506155 26882 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.562202 26551 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.558843 26718 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.572945 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:29.667611 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94742.4us
I20240426 13:19:29.565091 26877 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.653523 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:29.572335 26817 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:29.600121 26505 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:29.664995 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.671885 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:29.672266 25103 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
W20240426 13:19:29.627985 26870 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20240426 13:19:29.577670 26765 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20240426 13:19:29.668623 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:29.678143 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 358996us
W20240426 13:19:29.652529 26773 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.929s user 0.010s sys 0.002s
I20240426 13:19:29.602156 26441 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.600935 26413 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:29.601431 26775 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:29.683683 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 476149us
I20240426 13:19:29.645768 26681 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.697763 24953 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:29.700960 24953 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:29.698172 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:29.703140 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 486666us
I20240426 13:19:29.666867 26882 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.703907 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:29.704151 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.707520 25099 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.699477 25101 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.671512 26789 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:29.674669 26505 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.708954 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.710114 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.695613 25026 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20240426 13:19:29.714653 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:29.714962 24956 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.019s sys 0.041s
I20240426 13:19:29.717528 24947 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:29.699064 26756 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.687633 26748 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:29.719147 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.664177 26707 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.720643 25623 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:29.721671 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.702180 24953 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:29.722280 24956 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.019s sys 0.041s
I20240426 13:19:29.739552 24953 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.705523 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fad756437243d2959f5998d6bf8309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 13:19:29.668100 26654 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.707315 26508 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.760342 26508 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.760784 26698 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.761379 26654 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.762158 26508 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.671160 26551 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.697643 26441 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.764138 26551 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.764747 26441 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.765650 24950 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:29.770756 26707 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.772174 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:29.774168 26738 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.979s user 0.010s sys 0.000s
I20240426 13:19:29.774725 25102 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:29.711827 26789 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.748068 26681 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:29.717036 26773 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.757056 26711 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:29.723394 26481 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:29.777699 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:29.778205 24953 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240426 13:19:29.795248 24953 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:29.795692 24953 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240426 13:19:29.796002 24953 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:29.796363 24953 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:29.724805 26887 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.796660 24953 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:19:29.897778 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 229026us
I20240426 13:19:29.807861 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:29.903000 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 95040.6us
I20240426 13:19:29.904750 25026 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240426 13:19:29.908507 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:29.800829 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62917.8us
I20240426 13:19:29.912197 25069 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.915936 25028 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:29.915799 25069 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:29.918331 24953 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:29.791373 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.926110 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.711437 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:29.740356 26413 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.797719 26481 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.787312 26711 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:29.957345 26481 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:29.958938 26698 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:29.958798 26711 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.959992 26698 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.780313 26713 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.963362 26713 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.753782 26717 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.965641 26890 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.966789 26903 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:29.966691 26890 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.727885 26748 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:29.741187 26756 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.968669 26748 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:29.970578 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.970942 26756 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.807575 26765 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.972129 26765 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.974150 26698 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.974718 26765 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:29.789324 26789 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:29.967756 26903 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:29.978451 26903 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.979048 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.730226 26861 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20240426 13:19:29.990140 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.990195 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.991158 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1666) took 1164 ms. Trace:
I20240426 13:19:29.673435 26877 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:29.848340 26883 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.160s user 0.001s sys 0.001s
I20240426 13:19:29.867363 26882 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:29.999423 25097 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:29.775264 26884 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.001328 26884 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.003010 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.004501 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 201) took 1212 ms. Trace:
W20240426 13:19:29.852392 26885 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20240426 13:19:30.005079 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 204) took 1201 ms. Trace:
I20240426 13:19:30.005688 25022 rpcz_store.cc:276] 0426 13:19:28.803417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.803536 (+ 119us) service_pool.cc:225] Handling call
0426 13:19:28.811378 (+ 7842us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.829394 (+ 18016us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.829435 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:28.829488 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:28.829531 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.004565 (+1175034us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.004659 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.004870 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40914816}
I20240426 13:19:30.006862 26877 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:30.006984 26912 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.006911 24947 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:29.942276 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:30.008584 26600 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:29.951712 26684 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:29.955099 26689 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.007740 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.025684 25622 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:30.072191 25617 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:29.739529 26829 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20240426 13:19:29.980579 26888 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.009560 26884 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:29.992427 25023 rpcz_store.cc:276] 0426 13:19:28.826394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.826616 (+ 222us) service_pool.cc:225] Handling call
0426 13:19:28.827126 (+ 510us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:28.994741 (+167615us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.990663 (+995922us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:29.990945 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366953856}
I20240426 13:19:29.982177 26681 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:29.987916 26906 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:29.989118 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:29.989833 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.991521 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:29.811585 26887 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.086266 26887 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.009223 26600 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.086450 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:30.026856 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 449) took 1928 ms (client timeout 2492 ms). Trace:
W20240426 13:19:30.087608 25624 rpcz_store.cc:269] 0426 13:19:28.098181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.098817 (+ 636us) service_pool.cc:225] Handling call
0426 13:19:30.026397 (+1927580us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.026659 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2241880960}
I20240426 13:19:30.088523 26600 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.089790 26888 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.091084 24949 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.093888 26906 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.083534 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.095830 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.081473 26738 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:30.098268 26738 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:30.099560 26738 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.100847 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.102583 26906 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.004148 25097 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:30.010221 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 202) took 1218 ms. Trace:
I20240426 13:19:29.981021 26718 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.104084 26689 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.108924 26689 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.113996 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.062045 26882 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.103848 26888 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.004949 25025 rpcz_store.cc:276] 0426 13:19:28.791646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.793555 (+ 1909us) service_pool.cc:225] Handling call
0426 13:19:28.828139 (+ 34584us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.828476 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.828506 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.834172 (+ 5666us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.477640 (+643468us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:29.477695 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.477782 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.004119 (+526337us) raft_consensus.cc:1671] finished
0426 13:19:30.004153 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.004340 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1908,"raft.queue_time_us":542,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":25825,"thread_start_us":643125,"threads_started":1,"wal-append.queue_time_us":794805}
I20240426 13:19:30.005106 25105 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:30.054049 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.140831 25620 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:30.021849 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74786.2us
I20240426 13:19:30.021327 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:29.981854 26913 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:30.010458 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 61346.2us
I20240426 13:19:30.013478 26912 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.054932 26654 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.097784 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:29.967607 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.109742 26777 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.009634 25028 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:30.012543 26707 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.146811 26707 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.147924 24948 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:30.149006 26707 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.121069 26601 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.152256 26654 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.121534 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.121752 26738 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.156312 26747 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:30.156718 26826 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.156803 26747 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.157483 26747 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:30.122217 26914 log.cc:924] Time spent T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.111s user 0.002s sys 0.000s
I20240426 13:19:30.154639 26505 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.122323 26900 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.161929 26505 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.162307 26900 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.122862 26890 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.123512 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1233) took 1365 ms. Trace:
I20240426 13:19:30.125890 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.165856 25097 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.165344 26913 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.166253 26900 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.104784 25029 rpcz_store.cc:276] 0426 13:19:28.792002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.792179 (+ 177us) service_pool.cc:225] Handling call
0426 13:19:30.009797 (+1217618us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400415fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.009978 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":917025536}
I20240426 13:19:30.167629 26865 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.126775 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.129926 26653 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:30.168398 26653 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.165366 25102 rpcz_store.cc:276] 0426 13:19:28.757842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.759086 (+ 1244us) service_pool.cc:225] Handling call
0426 13:19:30.123147 (+1364061us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.123336 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2334134528}
I20240426 13:19:30.169530 26653 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.170953 26653 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.174722 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.083281 26684 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.177343 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.177122 26684 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.179452 26666 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:30.131054 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 407266us
I20240426 13:19:30.182521 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.169150 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.184798 25620 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.186285 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.187768 26912 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:30.189220 26879 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.623s user 0.003s sys 0.000s
I20240426 13:19:30.188845 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.191805 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.192087 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.192605 25025 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.129647 26789 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.125926 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1356) took 1280 ms. Trace:
I20240426 13:19:30.139123 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:30.140996 26864 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:19:30.076593 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 452) took 1005 ms. Trace:
I20240426 13:19:30.197917 25104 rpcz_store.cc:276] 0426 13:19:28.845032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.845233 (+ 201us) service_pool.cc:225] Handling call
0426 13:19:28.917748 (+ 72515us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:29.302971 (+385223us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.125518 (+822547us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.125762 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":861890816}
I20240426 13:19:30.197069 26872 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.204865 26872 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.206154 26852 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.206867 26852 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.207427 26872 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.152923 26601 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.154973 25623 raft_consensus.cc:2354] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
W20240426 13:19:30.155473 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 205278us
I20240426 13:19:30.155627 26654 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.174911 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 271) took 6263 ms. Trace:
I20240426 13:19:30.182633 26799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:44520 (request call id 290) took 3421 ms. Trace:
I20240426 13:19:30.132639 26689 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.134590 24949 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.142300 25099 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.201368 26562 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:30.201754 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 489) took 1741 ms (client timeout 1633 ms). Trace:
I20240426 13:19:30.202066 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 222) took 1692 ms. Trace:
I20240426 13:19:30.202734 26773 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.202080 26816 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.203403 26748 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.199697 25622 rpcz_store.cc:276] 0426 13:19:29.070854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:29.070989 (+ 135us) service_pool.cc:225] Handling call
0426 13:19:30.024973 (+953984us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:30.025312 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:30.075581 (+ 50269us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.076541 (+ 960us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24290048}
I20240426 13:19:30.203986 26696 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:30.207813 26871 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.733s user 0.002s sys 0.000s
I20240426 13:19:30.208711 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.210309 24996 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.224166 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:30.216917 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 220) took 1714 ms (client timeout 1984 ms). Trace:
I20240426 13:19:30.217283 26562 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.220019 26878 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.752s user 0.000s sys 0.003s
W20240426 13:19:30.220135 25616 rpcz_store.cc:269] 0426 13:19:28.459785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.462521 (+ 2736us) service_pool.cc:225] Handling call
0426 13:19:30.201411 (+1738890us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.201565 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":836499712}
I20240426 13:19:30.222790 26707 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.223616 26601 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.167676 26913 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.262615 26913 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.263209 26913 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.265157 25098 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20240426 13:19:30.265336 26913 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.234937 26852 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.265898 26721 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.263867 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1353) took 1589 ms. Trace:
I20240426 13:19:30.234158 26773 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.266474 26721 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.236006 24901 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ece98ef0f07f4184bedf2cb96b628f20, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:30.267827 24901 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ece98ef0f07f4184bedf2cb96b628f20, TS 46106ff992984d33bb4a4d8f30162639 -> TS 1590f80bdade41c6a755b8501c55fe16 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.231500 26816 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.266882 25100 rpcz_store.cc:276] 0426 13:19:28.674353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.674534 (+ 181us) service_pool.cc:225] Handling call
0426 13:19:28.720020 (+ 45486us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.754644 (+ 34624us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.754685 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.755164 (+ 479us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.164846 (+409682us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:29.164905 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.164997 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.219097 (+1054100us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:30.263352 (+ 44255us) raft_consensus.cc:1671] finished
0426 13:19:30.263372 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.263699 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34182,"raft.queue_time_us":36770,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":6613,"spinlock_wait_cycles":465287936,"thread_start_us":392119,"threads_started":1,"wal-append.queue_time_us":641359}
I20240426 13:19:30.268553 26816 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.240231 26696 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.241241 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.241231 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.241994 26789 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.242695 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.248623 25030 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240426 13:19:30.278345 26688 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.252537 26747 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.320168 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.320394 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.212785 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1611) took 1133 ms. Trace:
I20240426 13:19:30.321388 26747 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.265900 26825 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.324059 24948 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:30.324394 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.223887 26713 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.215654 26799 rpcz_store.cc:276] 0426 13:19:26.765217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.765367 (+ 150us) service_pool.cc:225] Handling call
0426 13:19:27.070272 (+304905us) ts_tablet_manager.cc:903] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
0426 13:19:29.296472 (+2226200us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:19:30.014282 (+717810us) tablet_metadata.cc:641] Metadata flushed
0426 13:19:30.014676 (+ 394us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:19:30.181366 (+166690us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:19:30.181821 (+ 455us) tablet_replica.cc:205] Creating consensus instance
0426 13:19:30.182267 (+ 446us) ts_tablet_manager.cc:958] Replica registered.
0426 13:19:30.182364 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":25453,"tablet-copy.queue_time_us":173608,"thread_start_us":1014304,"threads_started":5}
I20240426 13:19:30.327142 26799 tablet_copy_client.cc:783] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 0 data blocks...
I20240426 13:19:30.327652 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.258733 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.343845 26773 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.260982 26562 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.260888 26699 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.344507 24950 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:30.344699 25028 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:30.345142 26699 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.238603 26679 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.348356 26679 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.348861 25105 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:30.350407 26679 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:30.275993 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 51363.1us
I20240426 13:19:30.351181 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.352169 25069 leader_election.cc:398] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
W20240426 13:19:30.256500 24955 rpcz_store.cc:269] 0426 13:19:28.500213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.500396 (+ 183us) service_pool.cc:225] Handling call
0426 13:19:30.214506 (+1714110us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.214648 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2165327104}
I20240426 13:19:30.352303 26816 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.215492 24951 rpcz_store.cc:276] 0426 13:19:28.509420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.510927 (+ 1507us) service_pool.cc:225] Handling call
0426 13:19:28.511303 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:28.511908 (+ 605us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:28.511948 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:28.512518 (+ 570us) log.cc:842] Serialized 26 byte log entry
0426 13:19:29.291099 (+778581us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:29.291158 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:29.291259 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:30.188352 (+897093us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:30.201604 (+ 13252us) raft_consensus.cc:1671] finished
0426 13:19:30.201626 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.201878 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":165989760,"thread_start_us":778241,"threads_started":1,"wal-append.queue_time_us":963587}
I20240426 13:19:30.278831 26688 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.279021 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.285902 26852 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:30.357183 26852 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.357609 26852 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.357856 26681 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.358264 26681 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.358598 26681 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.286480 26656 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.287438 25617 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 2: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:30.288727 26916 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20240426 13:19:30.360213 25025 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.361074 26656 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.361462 26656 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.361820 26656 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.290067 26817 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:30.367707 26717 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.352455 25069 leader_election.cc:401] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:30.369776 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.370524 26713 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4.
I20240426 13:19:30.370826 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.371071 26713 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.295024 26696 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:30.295603 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 85400.8us
I20240426 13:19:30.296226 26903 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.297780 26721 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.298555 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.211783 24853 rpcz_store.cc:276] 0426 13:19:23.911498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:24.038475 (+126977us) service_pool.cc:225] Handling call
0426 13:19:30.174697 (+6136222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:27.356877 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:27.378893 (+ 22016us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:27.378932 (+ 39us) write_op.cc:427] Acquired schema lock
0426 13:19:27.378948 (+ 16us) tablet.cc:660] Decoding operations
0426 13:19:27.379368 (+ 420us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:27.379428 (+ 60us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:27.379463 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0426 13:19:27.383094 (+ 3631us) tablet.cc:699] Row locks acquired
0426 13:19:27.383115 (+ 21us) write_op.cc:260] PREPARE: finished
0426 13:19:27.383273 (+ 158us) write_op.cc:270] Start()
0426 13:19:27.383385 (+ 112us) write_op.cc:275] Timestamp: P: 1714137567383222 usec, L: 0
0426 13:19:27.383405 (+ 20us) op_driver.cc:331] REPLICATION: starting
0426 13:19:27.385603 (+ 2198us) log.cc:842] Serialized 912 byte log entry
0426 13:19:28.101765 (+716162us) op_driver.cc:444] REPLICATION: finished
0426 13:19:28.452363 (+350598us) write_op.cc:300] APPLY: starting
0426 13:19:28.460895 (+ 8532us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:28.464956 (+ 4061us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:28.476831 (+ 11875us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:28.553719 (+ 76888us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:28.586949 (+ 33230us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0426 13:19:28.587025 (+ 76us) write_op.cc:311] APPLY: finished
0426 13:19:28.587221 (+ 196us) log.cc:842] Serialized 29 byte log entry
0426 13:19:28.587398 (+ 177us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:28.587488 (+ 90us) write_op.cc:446] Released schema lock
0426 13:19:28.587533 (+ 45us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:28.587832 (+ 299us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:28.587868 (+ 36us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":855745,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":304000,"apply.run_cpu_time_us":2914,"apply.run_wall_time_us":160599,"num_ops":2,"prepare.queue_time_us":289666,"prepare.run_cpu_time_us":29185,"prepare.run_wall_time_us":328618,"raft.queue_time_us":262088,"raft.run_cpu_time_us":38601,"raft.run_wall_time_us":303119,"replication_time_us":711523,"thread_start_us":741837,"threads_started":4,"wal-append.queue_time_us":341091,"wal-append.run_cpu_time_us":30173,"wal-append.run_wall_time_us":1866685}]]}
I20240426 13:19:30.303309 25070 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.304459 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.308099 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:30.309363 26867 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240426 13:19:30.374305 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75901.4us
I20240426 13:19:30.309787 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.310662 26839 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.310879 26748 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.313542 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.314486 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.318459 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.258410 25026 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.394372 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.321445 24954 rpcz_store.cc:276] 0426 13:19:29.074817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:29.075913 (+ 1096us) service_pool.cc:225] Handling call
0426 13:19:29.076259 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:29.204088 (+127829us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:29.204130 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:29.204170 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:29.204200 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.198395 (+994195us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540004b354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.208135 (+ 9740us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:30.208334 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293524352}
I20240426 13:19:30.394975 25623 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:30.328819 26413 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.279239 26481 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.397102 26481 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.397910 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.398908 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.399547 25024 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.399435 26481 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.400527 26481 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:30.335487 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 196) took 1914 ms. Trace:
I20240426 13:19:30.400878 26481 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.400991 25030 rpcz_store.cc:276] 0426 13:19:28.420646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.420784 (+ 138us) service_pool.cc:225] Handling call
0426 13:19:28.421133 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:30.278009 (+1856876us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.335153 (+ 57144us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54004169d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:30.335418 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4201729792}
I20240426 13:19:30.400909 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:30.401919 26481 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.392306 26852 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.405923 26852 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.336732 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1229) took 1580 ms. Trace:
I20240426 13:19:30.338461 26711 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.327826 26799 tablet_copy_client.cc:647] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Starting download of 1 WAL segments...
W20240426 13:19:30.356097 26707 consensus_queue.cc:1159] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 5 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.2-4.2]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:30.356648 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 77750.6us
W20240426 13:19:30.288806 26880 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.790s user 0.002s sys 0.000s
I20240426 13:19:30.359781 26688 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.355367 26816 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.376368 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.375463 26825 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.415588 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.415925 26825 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.416345 26825 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.416648 26791 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.418224 26791 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.418951 26791 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:30.377086 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69219.1us
I20240426 13:19:30.376814 26839 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.420605 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.421669 26839 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.377501 26849 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.422317 26849 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.422591 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.422878 26859 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.355556 26893 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:19:30.423444 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.377945 25620 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.379798 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.425086 25025 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.372747 26903 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.426086 26903 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.427196 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.375384 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.428290 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:30.380249 26747 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:30.428961 26747 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:30.430145 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53833.6us
I20240426 13:19:30.429841 26747 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.380988 25616 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:30.382622 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.432801 26679 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.433097 25624 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:30.434218 26688 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.434834 26688 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.383734 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.385017 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.386024 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.391515 26789 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:30.406484 25098 rpcz_store.cc:276] 0426 13:19:28.756563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:28.758548 (+ 1985us) service_pool.cc:225] Handling call
0426 13:19:30.264700 (+1506152us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:30.265603 (+ 903us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:30.336529 (+ 70926us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54001b3350. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:30.336665 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3463254784}
I20240426 13:19:30.422713 26849 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.438314 26859 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:30.439083 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.423568 26893 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20240426 13:19:30.378638 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75441.5us
I20240426 13:19:30.436872 26688 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.380160 26656 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.419850 26481 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.450269 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.452495 25624 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:30.406499 26721 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.454002 26713 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:30.461028 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.470976 25027 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.461923 26791 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.435865 26825 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.476303 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.463469 26903 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.473546 25024 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:30.481204 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.484192 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.484282 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.370932 26681 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:30.487958 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72591.9us
W20240426 13:19:30.487932 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 72460.9us
I20240426 13:19:30.495865 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496022 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496254 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.496829 24955 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.498653 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.498775 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.499217 24955 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:30.499855 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.471858 26839 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:30.500638 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.474021 26721 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.505718 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.478714 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.515403 24953 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.484817 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.518237 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.518592 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:30.522575 24948 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.522751 25097 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:30.522065 24951 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.519088 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.521463 25619 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:30.525367 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.525518 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.515375 25105 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:30.524645 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.528179 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.528791 25099 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:30.529603 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.510694 26721 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.532230 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.534314 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.532550 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.532907 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.534745 25024 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:30.505491 26713 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:30.506842 26839 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.514701 26748 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:30.531737 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.540207 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.540293 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.536307 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.538715 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.538192 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:30.543509 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:30.485342 26849 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.535225 25617 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.545859 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.545121 25025 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.549518 26858 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:30.549849 26894 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.550259 25101 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.550709 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.551308 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.547243 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.586215 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.551466 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.556507 25098 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:30.546537 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.557143 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.561796 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.577653 25103 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.563807 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.519520 26893 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20240426 13:19:30.547029 26839 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:30.591315 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.589285 24955 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:30.592288 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.592968 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.594000 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:30.597529 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.546133 26689 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.604480 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.608505 26816 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:30.609573 26900 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:30.605620 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.580255 26858 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:30.604282 26865 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.585965 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:30.590543 26748 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:30.582867 26738 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:30.600402 26848 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.624825 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.620971 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.544873 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.589850 26789 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:30.637830 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.638062 25099 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:30.638657 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.640748 25029 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.641809 25103 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:30.642539 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:30.628414 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 163413us
W20240426 13:19:30.650846 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56894.6us
I20240426 13:19:30.557508 26747 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.592988 26717 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:30.606381 26689 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.634243 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.651938 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.628053 26852 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.654018 25105 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.645566 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.658277 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.635345 26848 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.661449 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.668103 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.665630 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.673142 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.666762 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.680151 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.681030 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.649997 26656 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.710109 25103 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.657258 26791 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.664011 26825 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.644268 26789 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.670805 25026 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.596202 26894 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.675109 26848 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.723094 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:30.793279 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54908us
I20240426 13:19:30.793989 25024 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:30.730448 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.794363 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.669539 26689 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.767424 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.746763 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.797722 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:30.798117 25624 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:30.799043 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.799714 26699 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.800192 26699 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.800554 26699 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.801502 26894 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.802932 24951 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:30.804157 25102 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:30.805224 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.805938 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:30.808521 26849 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:30.810783 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.671418 26653 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:30.719302 26717 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.754244 26713 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:30.813066 25104 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.680688 26681 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.722683 26666 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.825866 26666 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.826260 26666 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.739373 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:30.739008 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.830148 25029 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
W20240426 13:19:30.833122 26921 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20240426 13:19:30.835988 26717 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.843863 25102 raft_consensus.cc:2354] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.845889 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:30.738731 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.671774 26903 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:30.719892 25097 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:30.798919 26707 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.739064 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.716396 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.848345 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.848395 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.870774 24921 leader_election.cc:401] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240426 13:19:30.880856 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58413.3us
I20240426 13:19:30.680351 26776 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.884405 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:30.885219 26562 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:30.886360 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.853238 26748 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.888051 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:30.888723 24954 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:30.890426 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:30.891281 26681 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.891635 26681 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.891968 26681 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:30.892447 26911 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:30.892462 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 125185us
I20240426 13:19:30.893888 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.857367 25027 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:30.897732 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.851953 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:30.739426 26773 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.911540 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:30.912966 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 114017us
I20240426 13:19:30.666023 26858 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.920287 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:30.712123 26826 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.926573 26848 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.928087 26824 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.929663 26757 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240426 13:19:30.930827 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 214528us
I20240426 13:19:30.968533 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:30.931298 26825 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.969420 26825 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.941876 26826 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.969854 26825 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.970054 26826 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.972545 26848 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:30.973865 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 234879us
I20240426 13:19:30.973920 26839 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.975162 24950 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:30.935708 26902 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:30.976558 26902 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:30.978277 26902 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:30.978896 26684 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:30.981555 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.982244 25029 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.950564 26656 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:30.984369 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:30.984544 26656 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:30.985008 26656 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:30.985172 25024 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:30.917924 26773 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:30.986251 26773 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.990562 24948 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20240426 13:19:30.990929 24948 pending_rounds.cc:76] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 1
I20240426 13:19:30.936163 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:30.937449 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 274715us
I20240426 13:19:30.952600 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:30.882201 26791 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.944360 24921 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.000159 24921 leader_election.cc:401] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:19:30.949156 26717 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.859797 26699 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:30.960862 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.018482 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 81062.7us
I20240426 13:19:31.018978 24954 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.944890 26688 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:30.964702 26413 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:30.965135 26849 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:30.965624 26894 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:30.904314 26776 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:30.966161 26872 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:31.025494 26852 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:30.967533 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:30.991132 24948 pending_rounds.cc:97] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.2
W20240426 13:19:31.024664 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72162.3us
I20240426 13:19:31.047816 24948 raft_consensus.cc:2866] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:30.984560 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:31.026041 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.048192 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:30.905079 26747 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.044374 26872 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.042132 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.044376 25030 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20240426 13:19:31.026124 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.050274 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.028980 26824 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.051741 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67322us
I20240426 13:19:31.021873 26413 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.058239 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.052852 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.051901 25103 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.022677 26688 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.030716 26913 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:31.030468 26839 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.046193 26776 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.067101 24996 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:31.072284 24996 leader_election.cc:401] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:31.062909 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.052385 26900 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.052335 25617 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:31.058480 26872 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.060925 26684 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:31.029013 26859 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240426 13:19:31.057746 26867 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.081144 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.030265 26717 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.061146 26688 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.080291 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.061642 26413 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.076134 26601 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240426 13:19:31.079789 26684 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.051239 26894 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.085659 24996 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.070796 26902 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.116304 26912 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.071709 26791 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:31.086886 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.097079 26824 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.100376 26562 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:31.084849 26867 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.105681 26681 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.107017 26699 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.107604 26684 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.108522 26894 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.071707 26911 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:31.101606 26666 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.156839 25029 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:31.150385 26721 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.570s user 0.002s sys 0.010s
W20240426 13:19:31.163452 26933 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20240426 13:19:31.151635 26867 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.166330 26711 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.166603 26653 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.167600 26707 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:31.148288 26562 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:31.187520 25030 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.164474 26791 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.172848 26913 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3.
I20240426 13:19:31.176447 26789 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.170662 26894 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.210309 25619 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:31.207310 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 179763us
I20240426 13:19:31.207221 26825 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.179500 26721 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.194006 26791 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.203212 26717 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.231323 26930 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.232632 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 2 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.235272 26773 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.237268 26721 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.237803 25100 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.263262 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:31.263975 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.204210 26656 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.267572 26656 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.245499 26791 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.273641 24853 catalog_manager.cc:5484] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.256714 26562 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.278213 24853 catalog_manager.cc:5484] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 4 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.257428 26747 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.229650 26894 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.283787 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.255060 26824 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.293190 26562 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.342113 26826 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.343883 25102 raft_consensus.cc:2354] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.347153 26826 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.308140 26861 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.309749 26666 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.253993 26930 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.320281 26681 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.324694 26824 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.325179 26825 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.333938 26858 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.335332 26852 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.338874 26413 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.339663 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.341156 26773 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.340972 26872 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.279894 26776 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.272715 26859 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.306391 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240426 13:19:31.362579 24953 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.000s sys 0.049s
W20240426 13:19:31.353379 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.041s sys 0.024s
W20240426 13:19:31.400337 24953 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s user 0.000s sys 0.050s
W20240426 13:19:31.400391 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.042s sys 0.024s
I20240426 13:19:31.348055 26689 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.375504 26711 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.398944 24853 catalog_manager.cc:5484] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.410344 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:31.414223 25096 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:31.418224 24853 catalog_manager.cc:5484] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:31.395769 26653 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.373760 26861 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.453203 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.414369 26711 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.454525 25619 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.426602 26653 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.466037 26688 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.469475 26861 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.426113 26684 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.472043 26684 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.470865 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.472064 26867 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.403059 26859 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.488943 26688 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.492827 26826 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.497257 26776 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.497655 26894 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.496960 26826 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.506242 25096 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:31.427058 26849 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20240426 13:19:31.508754 26689 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.509589 26872 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.510396 26413 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:31.517856 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 462491us
I20240426 13:19:31.510627 26826 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.463771 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.512614 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:31.519243 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 502842us
W20240426 13:19:31.520478 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 425520us
I20240426 13:19:31.523415 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.467952 26791 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.527763 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 440979us
I20240426 13:19:31.535137 24920 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.536293 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.538976 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.541266 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.526459 26872 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:31.542991 25102 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.588829 26872 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:31.589713 26872 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.590926 26872 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.571044 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.595927 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.596441 25102 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:31.544147 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:31.545588 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 496115us
W20240426 13:19:31.533857 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 471046us
I20240426 13:19:31.548861 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.600459 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.599952 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.604048 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.604645 25027 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.549291 26867 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.605450 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.552366 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.544957 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.600484 25623 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.606258 25102 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:31.606704 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61825.1us
W20240426 13:19:31.606861 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62786.7us
I20240426 13:19:31.553799 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:31.530349 25617 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.554755 25100 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:31.542716 26656 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.555701 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.556193 26562 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.557065 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.613476 25103 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.558027 26913 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.559176 25026 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:19:31.559762 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.562741 26799 tablet_copy_client.cc:523] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240426 13:19:31.562629 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.535876 26894 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:31.566536 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:31.599095 26839 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.003s sys 0.024s
I20240426 13:19:31.568166 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.569221 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.569578 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.571038 26942 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240426 13:19:31.621830 24948 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.571466 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.622470 25099 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.571584 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.574555 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.622987 24951 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.576480 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.577538 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.623703 26894 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.625909 26894 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:31.626494 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.627053 25100 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.619632 26852 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:31.632651 26852 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:31.633069 26852 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.604586 26900 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.634070 26900 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.635816 26900 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.612215 26666 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:31.636884 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.637857 25025 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.638752 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:31.639035 25069 consensus_peers.cc:485] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 -> Peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069): Couldn't send request to peer 1590f80bdade41c6a755b8501c55fe16. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240426 13:19:31.640756 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.580214 26930 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.534147 26413 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:31.584124 26824 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:31.654743 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.655062 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.585508 26791 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.610849 26848 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.605494 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.537813 26938 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.543743 26717 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:31.607993 24920 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.609966 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.611779 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:31.676522 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68612.1us
I20240426 13:19:31.614328 25098 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.677116 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.616726 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.676451 25029 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:31.677510 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.616623 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.612762 25104 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:31.617966 24954 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.568018 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.680538 24950 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.624022 25620 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:31.578451 25619 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.643359 25100 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:31.622215 26707 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:31.643790 24951 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:31.608949 26935 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:31.703910 24952 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.642302 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:31.602814 26653 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.620136 26562 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:31.689113 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47467.4us
I20240426 13:19:31.689684 25096 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.657763 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.688686 26894 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.688741 25616 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:31.617061 26913 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.689216 26948 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.646041 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:31.716809 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 70812us
I20240426 13:19:31.704427 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.676090 26938 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.718034 24954 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:31.705950 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.718871 26946 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.719396 26946 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:31.636904 26865 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20240426 13:19:31.688818 26413 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:31.709844 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.707825 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.641862 26867 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20240426 13:19:31.619905 26911 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20240426 13:19:31.715031 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.718004 26938 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.703336 26848 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:31.729727 25619 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.720227 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.730252 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.704025 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:31.705617 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93881.4us
I20240426 13:19:31.731400 24951 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.731956 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.718720 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.721992 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.703135 26930 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:31.723385 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.708740 24949 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240426 13:19:31.703640 26707 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:31.726478 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.737952 24921 leader_election.cc:398] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.738256 24954 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 5
I20240426 13:19:31.739180 24921 leader_election.cc:398] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4.
I20240426 13:19:31.727734 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.727839 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.710888 26952 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.716980 26935 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:31.739584 25102 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:31.748082 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.740859 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.750378 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.751065 24954 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:31.751243 25102 raft_consensus.cc:2354] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:31.743300 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.748828 25103 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:31.753012 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.753803 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.755064 24955 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:31.752781 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1255) took 1120 ms. Trace:
I20240426 13:19:31.756340 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.756583 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.756794 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.757215 24954 raft_consensus.cc:2354] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:31.757755 24951 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.756381 25101 rpcz_store.cc:276] 0426 13:19:30.632609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.718939 (+ 86330us) service_pool.cc:225] Handling call
0426 13:19:30.763809 (+ 44870us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:30.892373 (+128564us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:30.892403 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:30.892833 (+ 430us) log.cc:842] Serialized 26 byte log entry
0426 13:19:31.272600 (+379767us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:31.272653 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:31.272733 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:31.752556 (+479823us) raft_consensus.cc:1671] finished
0426 13:19:31.752580 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:31.752720 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2628,"raft.queue_time_us":3120,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":18579,"thread_start_us":379355,"threads_started":1,"wal-append.queue_time_us":619072}
I20240426 13:19:31.764791 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.735543 26791 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:31.765740 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.766042 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.766726 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.767349 24954 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:31.736660 26861 raft_consensus.cc:2741] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240426 13:19:31.771394 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.769955 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:31.772219 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.741652 26707 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:31.772490 25103 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.741708 26912 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.764278 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.778633 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.778887 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:31.773941 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.776533 25099 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:31.777513 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.743705 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:31.760835 26666 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.781723 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:31.782863 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:31.729045 26684 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240426 13:19:31.792735 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.803110 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:31.762740 26931 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.810107 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:31.827493 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.762899 26826 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:31.835366 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80299.4us
I20240426 13:19:31.836133 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:31.836215 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.836887 25027 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:31.842514 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.843322 25024 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:31.770179 26791 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:31.774928 26748 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:31.741672 26930 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.835731 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:31.833465 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:31.842209 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:31.757764 26839 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20240426 13:19:31.849722 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.742233 26950 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.098s
I20240426 13:19:31.787039 26773 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:31.787101 26656 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:31.894455 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:31.896878 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60841.7us
I20240426 13:19:31.897600 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:31.778152 26894 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.915814 24949 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:31.929134 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95804.3us
I20240426 13:19:31.781349 26949 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:31.936576 25069 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:31.996904 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147274us
I20240426 13:19:31.997920 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.998566 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 13:19:31.999138 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62621.9us
I20240426 13:19:31.853037 26791 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
W20240426 13:19:31.918536 26934 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20240426 13:19:31.927989 26689 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.835747 26826 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.001832 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 169292us
I20240426 13:19:31.928687 26679 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:31.835181 25616 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:31.849269 26931 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:32.042412 25588 leader_election.cc:398] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:32.060932 25588 leader_election.cc:401] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:31.841435 26852 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.842139 26825 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:31.996703 26684 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.002413 26717 raft_consensus.cc:2741] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240426 13:19:32.092669 24953 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:32.050110 26562 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.050643 26826 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.055349 26967 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:32.051522 26653 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:31.917068 26848 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.080556 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 78133.5us
I20240426 13:19:31.930465 26711 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.074956 26679 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:32.079988 26689 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:31.934757 26773 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.115828 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 123847us
W20240426 13:19:32.113582 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 174152us
I20240426 13:19:32.000380 26949 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:31.934942 26748 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:31.895043 26413 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.099414 26913 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.102870 26666 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:32.140705 25623 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.008s
I20240426 13:19:32.094476 26684 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
W20240426 13:19:32.136909 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 139092us
W20240426 13:19:32.164563 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 122378us
I20240426 13:19:32.177058 25623 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:32.177651 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:31.937687 26946 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.100339 26912 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.134790 26951 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:32.111743 26562 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.138872 24949 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.113363 26964 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.001817 26935 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.189941 24950 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:32.107462 26967 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.226334 26967 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.227591 26967 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:32.228307 26944 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240426 13:19:32.206008 26562 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.230580 26562 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.231232 26562 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.191095 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.139729 26689 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:32.208920 26957 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
W20240426 13:19:32.211964 25098 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.523s user 0.013s sys 0.000s
I20240426 13:19:32.253829 25098 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:32.214083 26858 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.818s user 0.007s sys 0.002s
I20240426 13:19:32.216156 26948 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.216601 26717 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:32.217304 26872 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.626s user 0.015s sys 0.000s
W20240426 13:19:32.221777 26959 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240426 13:19:32.222406 26968 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3.
I20240426 13:19:32.196799 26931 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.224573 26852 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.225080 26748 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.235486 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44449.9us
I20240426 13:19:32.253294 26681 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:32.256244 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:32.259583 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:32.206617 26684 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.271014 25624 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240426 13:19:32.231775 26562 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.273685 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:32.277537 25623 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.239830 26949 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:32.285768 26684 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.287578 26978 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.277686 25624 pending_rounds.cc:76] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 5
I20240426 13:19:32.288214 25624 pending_rounds.cc:97] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240426 13:19:32.288611 25624 raft_consensus.cc:2866] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:32.288161 26978 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.290578 26978 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.291515 26938 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.000s
I20240426 13:19:32.291121 26912 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.298657 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135647us
I20240426 13:19:32.271569 26858 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.237869 25025 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:32.282743 26930 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.280411 26852 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:32.303175 26681 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.325064 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.270035 26717 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.284178 26711 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:32.323926 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.028s sys 0.025s
I20240426 13:19:32.334282 25029 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:32.343499 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s user 0.029s sys 0.025s
W20240426 13:19:32.326954 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.000s sys 0.006s
W20240426 13:19:32.344120 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 561365us
W20240426 13:19:32.344393 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.001s sys 0.006s
I20240426 13:19:32.344798 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.233688 26679 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.347503 25623 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:32.330940 26941 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.812s user 0.003s sys 0.000s
I20240426 13:19:32.291932 26938 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.353462 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 175900us
I20240426 13:19:32.354768 24996 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.358769 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.359226 24921 leader_election.cc:401] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:32.310592 26825 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240426 13:19:32.334071 26681 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.367245 25031 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.006s sys 0.009s
I20240426 13:19:32.363463 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:32.344920 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 572772us
I20240426 13:19:32.371989 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:32.376855 25031 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.529s user 0.006s sys 0.009s
I20240426 13:19:32.353456 26711 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.384384 25620 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:32.385783 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.387112 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.387594 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:32.388259 25623 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.387873 25096 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:32.362038 26859 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.376379 26681 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.391319 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.364526 26984 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:32.394110 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1258) took 1686 ms. Trace:
I20240426 13:19:32.406558 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.407238 25101 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:32.366494 26965 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:32.398500 26857 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
W20240426 13:19:32.409498 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46118us
I20240426 13:19:32.346729 26852 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.418552 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.417017 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.321807 26666 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.378826 26949 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.342223 26689 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.432618 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.426288 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:32.433701 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51217.2us
I20240426 13:19:32.427109 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.434476 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.405448 25105 rpcz_store.cc:276] 0426 13:19:30.707419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.811873 (+104454us) service_pool.cc:225] Handling call
0426 13:19:30.828545 (+ 16672us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:30.880284 (+ 51739us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:30.880335 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:30.899956 (+ 19621us) log.cc:842] Serialized 26 byte log entry
0426 13:19:31.162826 (+262870us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:31.162870 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:31.162949 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:32.292712 (+1129763us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:32.393846 (+101134us) raft_consensus.cc:1671] finished
0426 13:19:32.393867 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.394060 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27312,"raft.queue_time_us":61478,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":4303616,"thread_start_us":234549,"threads_started":1,"wal-append.queue_time_us":534891}
I20240426 13:19:32.423740 26984 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.435451 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.437206 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.423785 26857 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:32.442831 26982 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:32.443442 26982 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:32.443363 26857 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.444532 26963 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20240426 13:19:32.446514 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap starting.
I20240426 13:19:32.447574 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.446421 26979 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.453267 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:32.389786 26711 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.436702 26984 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.437590 26965 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:32.440164 26978 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.461323 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.465350 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.467541 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.467703 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.467322 26562 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.470634 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.471233 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.472788 26789 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:32.473760 25616 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:32.463913 26748 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:32.475593 26987 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.475844 26938 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.476181 26987 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.471515 25098 raft_consensus.cc:2379] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:32.448630 26974 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:32.451892 26949 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:32.457383 26979 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:32.490468 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:32.548238 25026 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:19:32.468669 26984 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.490324 26684 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.489979 26974 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.505182 26979 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.552403 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.480294 26912 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:32.502197 26949 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.451608 24956 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d8fda82f95348219809212d99d6704e"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:35216
I20240426 13:19:32.560742 24956 raft_consensus.cc:595] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.561108 24956 raft_consensus.cc:603] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:32.559259 24921 leader_election.cc:398] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:32.554895 26962 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.556293 26684 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.558924 26974 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:32.564103 26949 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:32.577741 26994 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:32.571481 26707 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.778s user 0.011s sys 0.004s
I20240426 13:19:32.563675 26413 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.444533 26982 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.570681 26717 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.636121 26987 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.623158 26684 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:32.633411 26989 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.167s user 0.000s sys 0.005s
I20240426 13:19:32.650985 26681 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.651804 25616 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:32.652318 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:32.657043 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:32.641599 26791 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.001s
I20240426 13:19:32.633512 26413 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:32.661707 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 89697.1us
I20240426 13:19:32.622073 26912 consensus_queue.cc:1046] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20240426 13:19:32.665623 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:32.666152 25028 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:32.696898 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77488.6us
I20240426 13:19:32.721508 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.722568 25025 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:32.646641 26852 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.586622 26962 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.640180 26707 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.669389 26413 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:32.734666 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 244) took 1010 ms. Trace:
I20240426 13:19:32.670620 26717 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:32.575757 26965 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:32.746850 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78184.6us
I20240426 13:19:32.748843 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:32.749063 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:32.725708 26711 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:32.757297 25105 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "499552c0877e46c5af41389ff1be3152"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:32.760192 26757 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.756309 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:32.762538 25028 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:32.765573 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:32.716069 26791 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.777848 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1361) took 1069 ms. Trace:
I20240426 13:19:32.778293 24954 rpcz_store.cc:276] 0426 13:19:31.708479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.837710 (+129231us) service_pool.cc:225] Handling call
0426 13:19:32.478145 (+640435us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.478302 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:32.478336 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.478375 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:32.478407 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.478458 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.777672 (+299214us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b54003c46d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:32.777786 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1472118016}
I20240426 13:19:32.781419 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 241) took 1058 ms. Trace:
I20240426 13:19:32.781847 24951 rpcz_store.cc:276] 0426 13:19:31.723458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.070844 (+347386us) service_pool.cc:225] Handling call
0426 13:19:32.095465 (+ 24621us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.167155 (+ 71690us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:32.167201 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.167260 (+ 59us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:32.167303 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.781173 (+613870us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b5400078fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:32.781221 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.781363 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164268416}
I20240426 13:19:32.793637 26791 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:32.796628 27005 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240426 13:19:32.799319 25022 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:32.801977 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:32.751834 26748 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:32.747134 26993 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:32.807500 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58753.7us
I20240426 13:19:32.810534 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:32.811277 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62311.9us
I20240426 13:19:32.812839 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:32.817310 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.833312 24950 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:32.798568 26987 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:32.747229 26978 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20240426 13:19:32.807360 26935 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:32.877789 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65035.6us
I20240426 13:19:32.878001 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:32.873888 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.872246 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.924527 24948 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:32.921037 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 160389us
I20240426 13:19:32.884166 26967 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:32.885835 26984 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.900899 25096 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:32.902247 25098 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:32.816939 26993 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.906903 26707 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:32.908630 27001 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20240426 13:19:32.760720 26757 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.760937 26717 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:32.764483 26852 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:32.926102 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:32.926375 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:32.969777 26994 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.037034 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.037110 26994 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:32.877038 26980 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:33.038832 26980 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.040103 26980 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.044143 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:32.940512 25101 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:32.880206 26987 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:32.940608 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.046288 26987 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:33.046885 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 106355us
I20240426 13:19:32.882421 26948 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:32.881093 26976 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:33.050670 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76911.9us
I20240426 13:19:32.819482 26748 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:32.957181 26952 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.003s sys 0.002s
I20240426 13:19:32.935825 26935 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:33.079399 26995 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20240426 13:19:33.098426 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:33.063406 26681 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.104822 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 130245us
I20240426 13:19:32.971511 26967 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.090510 26948 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.110636 25096 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.113685 26967 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.115660 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.115072 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.079089 26952 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.082s user 0.003s sys 0.004s
I20240426 13:19:33.149784 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.153846 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 242) took 1430 ms (client timeout 1627 ms). Trace:
W20240426 13:19:33.162242 24955 rpcz_store.cc:269] 0426 13:19:31.723713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.727543 (+ 3830us) service_pool.cc:225] Handling call
0426 13:19:33.153615 (+1426072us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.153784 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33842944}
I20240426 13:19:33.163829 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.100622 26931 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.175714 24952 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:33.066963 26852 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.090134 26976 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.110582 27023 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:33.180914 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144014us
I20240426 13:19:33.154882 26969 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.144775 27013 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.178627 26930 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.201449 24951 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:33.201853 24951 pending_rounds.cc:76] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:33.202102 24951 pending_rounds.cc:97] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20240426 13:19:33.187721 26959 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240426 13:19:33.202756 24951 raft_consensus.cc:2866] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:33.198504 27028 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.198042 26969 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.179108 27015 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.270855 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.200539 27014 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.230s user 0.000s sys 0.000s
I20240426 13:19:33.199800 26748 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.273177 27022 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240426 13:19:33.275420 24955 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:33.110162 26948 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.186270 26962 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.310017 26962 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:33.195185 26994 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.277122 26984 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.276770 26930 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.310947 26994 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.313109 26994 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.314767 26948 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.277917 27028 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.280256 26969 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.280673 26992 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.218s
I20240426 13:19:33.281570 27004 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:33.350482 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1358) took 1644 ms. Trace:
I20240426 13:19:33.351063 24952 rpcz_store.cc:276] 0426 13:19:31.706195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.826927 (+120732us) service_pool.cc:225] Handling call
0426 13:19:33.154230 (+1327303us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.181155 (+ 26925us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:33.350206 (+169051us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400387a50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.350402 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3051885824}
I20240426 13:19:33.353034 26993 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.320986 26999 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.689s user 0.002s sys 0.000s
I20240426 13:19:33.289105 27015 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.290350 26721 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:33.206863 27006 log.cc:924] Time spent T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.397s user 0.000s sys 0.001s
I20240426 13:19:33.274639 26413 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.206751 27013 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.201970 27023 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:33.384745 24955 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:33.388986 26684 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.729s user 0.009s sys 0.000s
I20240426 13:19:33.404183 27013 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.391700 26976 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.403993 26684 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.354277 26757 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.295353 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87644.4us
I20240426 13:19:33.299700 27020 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:33.300855 27010 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.221s user 0.000s sys 0.001s
I20240426 13:19:33.182917 26931 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.415417 26684 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:32.741647 24949 rpcz_store.cc:276] 0426 13:19:31.724221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.206949 (+482728us) service_pool.cc:225] Handling call
0426 13:19:32.207506 (+ 557us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.236000 (+ 28494us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:32.550981 (+314981us) log.cc:842] Serialized 14 byte log entry
0426 13:19:32.734422 (+183441us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:32.734475 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:32.734503 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:32.734549 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:32.734626 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84794,"prepare.queue_time_us":403679,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1891,"raft.queue_time_us":160551,"thread_start_us":375529,"threads_started":3}
I20240426 13:19:33.205385 27016 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.323191 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 502) took 1609 ms. Trace:
I20240426 13:19:33.417984 26931 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.332537 27024 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20240426 13:19:33.333884 27028 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.333268 26902 raft_consensus.cc:3032] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:33.335453 26930 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:33.340250 27021 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
W20240426 13:19:33.398967 26979 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.834s user 0.009s sys 0.000s
I20240426 13:19:33.408357 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.389178 27017 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20240426 13:19:33.431195 24955 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.410717 26413 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.388716 26983 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.354785 26717 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.389729 26852 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.390372 26935 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:33.395608 26974 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.765s user 0.008s sys 0.000s
I20240426 13:19:33.413532 27023 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.413678 26987 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.414848 26711 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.416113 26656 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:33.417495 25620 rpcz_store.cc:276] 0426 13:19:31.713727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.713967 (+ 240us) service_pool.cc:225] Handling call
0426 13:19:31.727236 (+ 13269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:32.384820 (+657584us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:33.322887 (+938067us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:33.323095 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1499101952}
W20240426 13:19:33.322382 24953 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1647) took 1753 ms (client timeout 2131 ms). Trace:
W20240426 13:19:33.288193 26826 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.170s user 0.025s sys 0.000s
I20240426 13:19:33.421267 26902 raft_consensus.cc:729] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:33.429812 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.419600 26721 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:33.396494 26949 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.762s user 0.010s sys 0.000s
I20240426 13:19:33.441612 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.434276 26931 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:33.476912 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47191.6us
I20240426 13:19:33.466962 24955 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:33.435839 27016 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.512337 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.474530 25624 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:33.513191 25022 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.513000 27016 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:33.510012 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 395144us
I20240426 13:19:33.511834 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.497443 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.497952 26893 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240426 13:19:33.504916 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.501528 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.497982 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.507184 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.497051 25028 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:33.507427 26825 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:33.519534 24955 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:33.511440 25031 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.511929 26721 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:33.446130 24953 rpcz_store.cc:269] 0426 13:19:31.568863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.568979 (+ 116us) service_pool.cc:225] Handling call
0426 13:19:33.322114 (+1753135us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:33.322297 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1075071232}
I20240426 13:19:33.510955 26747 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.524356 26747 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.469985 26656 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.433167 26979 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.522567 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.516571 26945 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:33.520395 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:33.526347 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.443212 26983 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.528553 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.529773 27016 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.531064 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.536602 27035 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.553850 24949 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:33.475145 26711 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.482452 26912 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.564662 27016 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:33.565459 26912 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.567255 26717 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:33.496501 26791 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.443020 26974 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.568310 26791 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:33.569542 26974 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.547695 24953 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:33.569240 26656 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.518529 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.549201 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.570575 26656 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.518867 26931 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:33.551681 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.552778 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.497020 26949 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.536993 25022 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:33.556130 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:33.483479 26826 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.564239 26852 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.566030 27016 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:33.550933 25029 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:33.517349 26992 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:33.547915 26902 consensus_queue.cc:260] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.701654 25623 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240426 13:19:33.763484 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.697258 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:33.572139 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.702180 25101 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:33.576831 25616 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:33.633261 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.763520 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.573945 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:33.696951 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:33.767571 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 215963us
I20240426 13:19:33.769953 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:33.765831 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.768944 25100 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:33.770813 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.768579 25097 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
W20240426 13:19:33.772177 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 256907us
I20240426 13:19:33.773167 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.774655 25030 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:33.703209 24955 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:33.713902 24948 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240426 13:19:33.718569 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 206812us
I20240426 13:19:33.544389 26945 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:33.779426 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.780755 24950 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.780233 26713 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:33.789320 27013 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:33.790050 26902 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.789847 27013 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.790665 27029 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.791772 24921 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.792493 24921 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:33.792763 27036 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.794199 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:33.795046 26982 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:33.767443 26935 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.795758 26935 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.796624 26987 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.798583 26711 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.799947 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:33.702877 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.545958 27034 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.259s
I20240426 13:19:33.801546 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.804061 26747 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.546239 26721 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.658771 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:33.814680 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44812.5us
I20240426 13:19:33.815686 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:33.719955 27005 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20240426 13:19:33.815769 24995 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:33.826670 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.827581 25030 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.570138 26791 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:33.820183 25105 raft_consensus.cc:595] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:33.772967 26969 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.817199 26681 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:33.828832 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.829517 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.699482 26983 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:33.832753 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.714738 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.830109 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.808916 26945 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:33.781651 27023 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.791847 27013 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.776221 27016 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:33.838551 25103 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:33.814940 26684 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.642743 26912 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.835191 25105 raft_consensus.cc:603] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:33.832010 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:33.844260 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.823452 26721 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.817659 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.881398 25024 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:33.860384 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.868366 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.882761 25022 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:33.867568 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.886647 25096 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:33.830569 26980 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.053s sys 0.079s
I20240426 13:19:33.828869 26747 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:33.854876 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:33.887543 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55876.9us
I20240426 13:19:33.888249 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.837137 26713 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:33.891892 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:33.791178 27029 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.896806 25617 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:33.897078 26713 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:33.905532 26713 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:33.890435 26980 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s user 0.057s sys 0.080s
I20240426 13:19:33.905663 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.906388 25617 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:33.906965 26980 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.907440 26980 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.842401 26979 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:33.890673 27023 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.886649 27031 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:33.909409 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:33.909034 27031 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.915092 27016 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:33.846850 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:33.896394 26852 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:33.886425 26839 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:33.925287 26852 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:33.927584 26988 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.464s user 0.005s sys 0.000s
I20240426 13:19:33.835861 27034 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20240426 13:19:33.889864 27050 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20240426 13:19:33.943784 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:33.865060 26949 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.907562 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:33.839051 26413 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240426 13:19:33.846675 26969 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:33.861094 27015 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.202142 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:34.204634 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:33.893813 26987 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:34.072948 25103 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.212214 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:33.948988 25026 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.207999 25030 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:34.128731 26799 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240426 13:19:34.221287 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:34.203199 26826 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.224678 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:34.226727 26982 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:34.186215 27029 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.227591 26982 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.227947 26982 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.228286 26982 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:34.226773 27044 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.334s user 0.000s sys 0.000s
I20240426 13:19:34.224766 27015 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:34.233008 27015 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:33.992442 26717 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.235487 25622 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.153849 26979 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.051510 26679 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:33.948782 27035 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.169818 26993 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.237046 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.177959 26983 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.239933 26949 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:34.073419 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.177299 26711 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.244072 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:33.889269 26684 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.272787 25623 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.271129 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.276804 24950 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:33.948601 26825 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.288s
I20240426 13:19:33.980809 26977 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.259s
W20240426 13:19:34.278038 24954 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1371) took 1468 ms (client timeout 1953 ms). Trace:
W20240426 13:19:34.299245 24954 rpcz_store.cc:269] 0426 13:19:32.810106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.810241 (+ 135us) service_pool.cc:225] Handling call
0426 13:19:34.247771 (+1437530us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b5400367d28. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda96290034 00007fda9525a0a7 00007fda95259838 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:34.277984 (+ 30213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17551232}
I20240426 13:19:34.050127 26681 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.190s
I20240426 13:19:34.279827 27036 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.239567 27013 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:34.226130 26826 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240426 13:19:34.245764 26980 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.295292 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.309245 26980 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.309430 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.309732 26980 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.310393 26413 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:34.310984 26413 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:34.311420 26413 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:34.311736 26980 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:33.900455 26931 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.298195 26975 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:34.312691 26975 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.312359 26413 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.313086 26975 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:34.298105 26711 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.315686 26980 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.316437 26711 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.277066 26950 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.317272 26950 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.281038 26852 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.306602 25027 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.209383 26987 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.298594 26721 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.306980 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.295934 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.323061 25026 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:34.316227 26980 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.306607 25024 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:34.245963 26717 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.268276 26979 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.286134 26949 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:34.324609 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197915us
I20240426 13:19:34.273576 26679 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.333865 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.278291 27035 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.281879 26993 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.333006 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.298938 25616 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:34.307807 27013 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:34.341785 25623 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:34.318802 26711 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.319229 27031 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.360366 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.366950 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.324880 26987 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.363988 25022 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 5
I20240426 13:19:34.299352 26983 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.370616 26993 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.374055 27035 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.375001 26679 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.375994 26950 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.375640 26711 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.376346 27051 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:34.377022 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.377205 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.521199 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.521519 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.367286 25589 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.522274 25102 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:34.522439 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:34.522531 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 155318us
I20240426 13:19:34.523149 25098 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:34.388717 26902 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.516904 26980 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.517534 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.385861 26975 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.550269 26902 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.390717 27051 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:34.520279 26825 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.552233 27051 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.552372 26825 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.554502 26825 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.552577 26902 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.521044 26717 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.523058 26983 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.524005 26993 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.556329 26993 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.558223 26993 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.524745 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.381350 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:34.559120 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 177839us
W20240426 13:19:34.560534 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.953s (SENT) [suppressed 3 similar messages]
I20240426 13:19:34.527467 26977 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.561517 25028 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.561741 26977 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.562474 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.563525 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.528120 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.522987 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.542665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.544155 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.527336 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.387128 26952 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.394539 26949 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.550405 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.560465 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:34.569149 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 44474.1us
I20240426 13:19:34.569262 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.545830 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.569276 25104 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:34.585655 25028 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.563970 26977 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.611073 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.609678 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.615527 25031 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:34.587918 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240426 13:19:34.564955 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:34.627214 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.624315 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.589632 26975 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.588342 26993 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.636812 26993 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:34.637255 26993 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:34.639720 26976 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:34.588188 27058 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:34.640251 26976 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:34.606973 26825 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:34.641256 26976 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.564582 26684 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.560469 26969 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:34.628501 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:34.626628 26936 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.632570 26977 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:34.633800 26711 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:34.653641 25023 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:34.655005 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:34.755527 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 131274us
W20240426 13:19:34.755669 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100809us
I20240426 13:19:34.656080 25026 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:34.756855 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:34.748525 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.748714 26949 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.764104 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:34.766184 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:34.770835 25097 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:34.773769 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.774236 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.774986 25070 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.776316 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:34.779820 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 219312us
W20240426 13:19:34.757576 26713 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.851s user 0.010s sys 0.000s
I20240426 13:19:34.784698 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:34.787647 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.748008 27058 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.796753 27065 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.566102 27013 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.748024 27066 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:34.763437 26976 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:34.626302 26982 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.804728 26824 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:34.779332 26980 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:34.768965 26711 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:34.816562 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:34.791646 26717 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:34.826982 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.757166 26987 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.797291 27035 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:34.780081 27067 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:34.837075 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.000s sys 0.070s
W20240426 13:19:34.840375 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.000s sys 0.070s
I20240426 13:19:34.819867 26799 tablet_bootstrap.cc:492] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Bootstrap complete.
I20240426 13:19:34.854087 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.854305 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.809082 27072 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:34.858664 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:34.855458 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.862185 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.863806 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.864714 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:34.799546 26713 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.867205 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.820407 26980 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:34.868291 25623 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:34.854871 26931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.006s sys 0.019s
I20240426 13:19:34.877615 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:34.835172 26824 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.879892 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.842530 27066 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:34.900462 25101 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:34.890446 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:34.848479 26711 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.856490 27036 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:34.853963 26979 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:34.801903 27058 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.878357 27065 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:34.878167 26980 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:34.877869 27072 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:34.877125 26717 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240426 13:19:34.888494 26931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.006s sys 0.019s
I20240426 13:19:34.927772 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 18) took 2185 ms. Trace:
I20240426 13:19:34.870842 27068 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:34.607136 26952 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:34.906419 26799 ts_tablet_manager.cc:1367] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent bootstrapping tablet: real 2.460s user 0.028s sys 0.029s
I20240426 13:19:34.898041 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.753995 26936 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:34.943436 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:34.937728 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:34.938556 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:34.940542 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.029767 24950 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.863616 27067 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20240426 13:19:34.940193 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:34.880633 26849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.009s sys 0.022s
I20240426 13:19:34.906159 27066 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:34.935213 26902 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:35.006626 26824 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.008947 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.047s sys 0.051s
W20240426 13:19:35.050403 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.048s sys 0.051s
W20240426 13:19:35.013350 27070 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20240426 13:19:35.016868 26945 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.156s user 0.008s sys 0.003s
I20240426 13:19:35.018179 26717 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240426 13:19:35.022079 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 219415us
I20240426 13:19:35.020874 25105 rpcz_store.cc:276] 0426 13:19:32.750571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.750701 (+ 130us) service_pool.cc:225] Handling call
0426 13:19:34.921932 (+2171231us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:34.922065 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2443024256}
I20240426 13:19:35.028427 26773 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.031198 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 323) took 1554 ms. Trace:
I20240426 13:19:34.940542 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:35.031569 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53497.7us
I20240426 13:19:35.039399 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.031858 24948 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:35.041694 24949 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
W20240426 13:19:35.050745 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 848245us
W20240426 13:19:35.051062 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.022s sys 0.040s
I20240426 13:19:35.056715 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.089732 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.023s sys 0.040s
I20240426 13:19:35.090270 24956 raft_consensus.cc:2379] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:35.023082 25026 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:35.001046 27065 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.060214 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:35.043989 26849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.009s sys 0.022s
I20240426 13:19:35.066220 25620 rpcz_store.cc:276] 0426 13:19:33.476545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.476687 (+ 142us) service_pool.cc:225] Handling call
0426 13:19:34.943255 (+1466568us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:35.031118 (+ 87863us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12377984}
I20240426 13:19:35.086238 24950 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:35.091915 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:35.114271 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:35.099046 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.103884 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43762.7us
I20240426 13:19:35.052759 26902 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.117491 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.046649 25616 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:34.964613 27036 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.105474 26839 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.106870 26799 raft_consensus.cc:348] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:35.107797 26893 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:35.102306 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:35.110415 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 241193us
I20240426 13:19:35.103863 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.057150 27067 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.115105 26982 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.062566 26945 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.127070 27070 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 13:19:35.118350 26858 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20240426 13:19:35.121434 26983 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:35.160833 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.161780 26980 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.162204 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.119546 26773 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.164192 26979 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.164410 26773 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.164670 26979 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.166327 26773 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.129403 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.166937 26969 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.167454 26969 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.162992 25102 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:35.169287 26969 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.061219 27035 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20240426 13:19:35.169960 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.170272 25624 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:35.170814 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 326) took 1657 ms. Trace:
I20240426 13:19:35.170877 27058 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.171088 27035 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:35.171525 27035 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:35.150868 24956 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:35.167670 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.110191 27065 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:35.122491 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.172358 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 228) took 2794 ms (client timeout 1954 ms). Trace:
I20240426 13:19:35.130292 27066 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.151640 24996 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:35.248315 24996 leader_election.cc:401] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20240426 13:19:35.249303 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97848.9us
I20240426 13:19:35.250299 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.154198 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1296) took 1173 ms. Trace:
I20240426 13:19:35.120647 26902 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:35.128674 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.251188 25022 rpcz_store.cc:276] 0426 13:19:33.984277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.984499 (+ 222us) service_pool.cc:225] Handling call
0426 13:19:35.154006 (+1169507us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.154134 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347513728}
W20240426 13:19:35.152184 27077 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20240426 13:19:35.199407 26993 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.562s user 0.005s sys 0.006s
I20240426 13:19:35.198844 27035 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:35.178643 25030 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:35.148802 26936 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.158808 26945 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.157219 26982 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.179050 26839 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.171644 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.180002 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:35.123345 26711 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.144268 26987 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.173192 26825 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.149931 26799 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Initialized, Role: LEARNER
I20240426 13:19:35.122749 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 330) took 1222 ms. Trace:
I20240426 13:19:35.287307 25617 rpcz_store.cc:276] 0426 13:19:33.898438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.907008 (+ 8570us) service_pool.cc:225] Handling call
0426 13:19:33.907270 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.907464 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:34.288864 (+381400us) log.cc:842] Serialized 14 byte log entry
0426 13:19:34.815220 (+526356us) log.cc:842] Serialized 14 byte log entry
0426 13:19:35.059604 (+244384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:35.059727 (+ 123us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:35.070772 (+ 11045us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.110078 (+ 39306us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.120510 (+ 10432us) inbound_call.cc:173] Queueing success response
0426 13:19:35.165401 (+ 44891us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x7b18001fdd40. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":27372,"prepare.queue_time_us":418073,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":154380,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":10079,"spinlock_wait_cycles":22158080,"thread_start_us":1041913,"threads_started":3,"wal-append.queue_time_us":552014}
I20240426 13:19:35.268795 26931 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.292573 26931 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.295153 26931 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.265004 27062 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:35.298883 27062 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.301277 27062 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.301879 26825 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.304455 26825 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.166344 26979 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.182878 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1389) took 1731 ms. Trace:
I20240426 13:19:35.306886 24953 rpcz_store.cc:276] 0426 13:19:33.451039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.570350 (+119311us) service_pool.cc:225] Handling call
0426 13:19:33.570606 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.638144 (+ 67538us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.638188 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.638243 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:33.638295 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.182593 (+1544298us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.182654 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.182813 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6837120}
I20240426 13:19:35.308751 26839 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.309432 26839 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.309962 26839 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.310863 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.152271 27067 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.184096 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1283) took 1612 ms. Trace:
I20240426 13:19:35.312388 25096 rpcz_store.cc:276] 0426 13:19:33.571872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.573647 (+ 1775us) service_pool.cc:225] Handling call
0426 13:19:35.183873 (+1610226us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.184017 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259534208}
I20240426 13:19:35.313400 27067 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.314126 25096 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:35.130584 26713 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:35.185982 26976 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:35.171985 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68202.6us
I20240426 13:19:35.186865 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1380) took 2371 ms. Trace:
I20240426 13:19:35.318087 24951 rpcz_store.cc:276] 0426 13:19:32.815092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.034165 (+219073us) service_pool.cc:225] Handling call
0426 13:19:33.034473 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.288686 (+254213us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.186603 (+1897917us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400366fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.186788 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383228160}
I20240426 13:19:35.188735 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 244) took 1083 ms. Trace:
I20240426 13:19:35.319303 25025 rpcz_store.cc:276] 0426 13:19:34.105758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.105929 (+ 171us) service_pool.cc:225] Handling call
0426 13:19:35.188060 (+1082131us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:35.188255 (+ 195us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.188424 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.188453 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:35.188496 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:35.188529 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.188583 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.188675 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11662080}
I20240426 13:19:35.189538 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:35.196239 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70644us
I20240426 13:19:35.321198 25028 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240426 13:19:35.193737 27030 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.671s user 0.007s sys 0.006s
I20240426 13:19:35.163739 26980 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.171186 25624 rpcz_store.cc:276] 0426 13:19:33.513823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.514602 (+ 779us) service_pool.cc:225] Handling call
0426 13:19:35.169909 (+1655307us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:35.170072 (+ 163us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.170616 (+ 544us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.170768 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5041920}
W20240426 13:19:35.122315 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 244804us
I20240426 13:19:35.329221 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:35.246212 25029 rpcz_store.cc:269] 0426 13:19:32.377809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:32.377936 (+ 127us) service_pool.cc:225] Handling call
0426 13:19:35.172152 (+2794216us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001059d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:35.172279 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2634617728}
I20240426 13:19:35.330701 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:35.149492 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.152274 27072 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:35.254684 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:35.331712 24949 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:35.255227 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.256639 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.332902 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.333544 24950 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:35.333777 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.334340 24956 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:35.129945 27036 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:35.265480 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.331257 25029 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:35.268913 25619 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 325) took 1756 ms. Trace:
I20240426 13:19:35.274653 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.277336 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:35.279120 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:35.257344 27038 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:35.259568 26912 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:35.277093 26977 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240426 13:19:35.268287 27061 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.283840 26945 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:35.284787 26679 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:35.286015 26799 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:35.308674 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.314896 26976 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:35.199510 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 333221us
I20240426 13:19:35.325906 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.326475 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.377606 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:35.362643 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:35.368499 27086 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.375301 24949 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:35.261610 26993 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:35.371312 26975 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:35.336354 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:35.362401 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 107793us
I20240426 13:19:35.368892 26979 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:35.375041 26978 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:35.336827 25619 rpcz_store.cc:276] 0426 13:19:33.512289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.512689 (+ 400us) service_pool.cc:225] Handling call
0426 13:19:35.179851 (+1667162us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:35.268867 (+ 89016us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9327488}
I20240426 13:19:35.323515 27066 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:35.394171 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:35.368366 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:35.376256 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:35.279924 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.028748 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:35.277731 26967 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.007s user 0.021s sys 0.018s
I20240426 13:19:35.275308 26982 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.030153 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.061208 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1294) took 2134 ms. Trace:
I20240426 13:19:36.031062 25624 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:35.666205 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 295231us
I20240426 13:19:36.061609 25027 rpcz_store.cc:276] 0426 13:19:33.897916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.898044 (+ 128us) service_pool.cc:225] Handling call
0426 13:19:33.898401 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.318923 (+420522us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.031844 (+1712921us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000b0c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:36.031992 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938320768}
I20240426 13:19:36.031661 25029 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.029567 24949 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:35.528196 27039 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.063685 24949 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.063580 27039 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.031929 25103 raft_consensus.cc:595] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.066128 27039 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:35.437134 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 252) took 2621 ms. Trace:
I20240426 13:19:36.067956 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1303) took 1242 ms. Trace:
I20240426 13:19:36.066859 24947 rpcz_store.cc:276] 0426 13:19:32.815520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.035890 (+220370us) service_pool.cc:225] Handling call
0426 13:19:33.054403 (+ 18513us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.078236 (+ 23833us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:33.509856 (+431620us) log.cc:842] Serialized 14 byte log entry
0426 13:19:33.768914 (+259058us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.768959 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:33.768994 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.423670 (+1654676us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:35.423757 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:35.437063 (+ 13306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148281,"prepare.queue_time_us":112979,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":131218,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"spinlock_wait_cycles":55118976,"thread_start_us":524430,"threads_started":2,"wal-append.queue_time_us":319100,"wal-append.run_cpu_time_us":1874,"wal-append.run_wall_time_us":1190160}
I20240426 13:19:36.068450 25103 rpcz_store.cc:276] 0426 13:19:34.825636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.067057 (+1241421us) service_pool.cc:225] Handling call
0426 13:19:36.067384 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.067572 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.067607 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.067660 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.067712 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.067775 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.067895 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.033568 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:36.025820 25620 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.008s sys 0.032s
W20240426 13:19:36.070310 25620 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.008s sys 0.032s
I20240426 13:19:36.034663 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1297) took 1816 ms. Trace:
I20240426 13:19:36.071293 25030 rpcz_store.cc:276] 0426 13:19:34.218200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.219330 (+ 1130us) service_pool.cc:225] Handling call
0426 13:19:35.172949 (+953619us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:35.259894 (+ 86945us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.034424 (+774530us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54001059d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:36.034587 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2192438528}
I20240426 13:19:35.361764 27036 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.074041 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1449) took 1457 ms. Trace:
I20240426 13:19:36.074291 25023 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:36.056422 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 718569us
I20240426 13:19:35.669252 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.151542 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 308) took 1544 ms. Trace:
I20240426 13:19:36.144665 25097 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:36.062155 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 311) took 1438 ms. Trace:
I20240426 13:19:36.152168 25098 rpcz_store.cc:276] 0426 13:19:34.616943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.062789 (+1445846us) service_pool.cc:225] Handling call
0426 13:19:36.073560 (+ 10771us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.073742 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.073768 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.073805 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.073838 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.073904 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.073999 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.153529 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 310) took 1537 ms. Trace:
I20240426 13:19:36.153407 25102 rpcz_store.cc:276] 0426 13:19:34.623705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061469 (+1437764us) service_pool.cc:225] Handling call
0426 13:19:36.061728 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.061875 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.061898 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.061935 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.061967 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.062016 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.062108 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:35.807083 26711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.003s sys 0.028s
I20240426 13:19:36.156773 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:35.377200 26965 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.030241 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1757) took 1137 ms. Trace:
W20240426 13:19:35.855255 27051 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.247s user 0.009s sys 0.001s
I20240426 13:19:36.153179 25105 rpcz_store.cc:276] 0426 13:19:34.607060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061414 (+1454354us) service_pool.cc:225] Handling call
0426 13:19:36.150997 (+ 89583us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.151201 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.151225 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.151263 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.151300 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.151363 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.151487 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.010030 27085 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.157493 24953 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:36.187183 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.187813 25623 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:36.028354 26987 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:36.188236 26982 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:36.188839 26902 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.930s user 0.010s sys 0.000s
I20240426 13:19:36.188707 26987 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.189239 26902 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.190052 26993 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.190619 26987 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.191051 26902 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.191164 26987 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:36.157095 26711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s user 0.005s sys 0.034s
I20240426 13:19:36.184367 26965 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.194061 27035 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.927s user 0.009s sys 0.000s
I20240426 13:19:36.194439 27035 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.181253 26979 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:36.195214 26965 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.060376 26983 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:36.197932 26987 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:19:36.169730 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107400us
I20240426 13:19:36.179512 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.226186 24956 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
W20240426 13:19:36.171295 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 137797us
I20240426 13:19:36.075244 26824 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:35.865967 26839 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.159194 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:36.179126 25026 rpcz_store.cc:276] 0426 13:19:34.892546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.125095 (+232549us) service_pool.cc:225] Handling call
0426 13:19:35.127254 (+ 2159us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:35.154368 (+ 27114us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.154399 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:35.154804 (+ 405us) log.cc:842] Serialized 26 byte log entry
0426 13:19:35.275881 (+121077us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:35.275925 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:35.275990 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:36.030056 (+754066us) raft_consensus.cc:1671] finished
0426 13:19:36.030076 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.030191 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22080,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":120721,"threads_started":1,"wal-append.queue_time_us":179749}
W20240426 13:19:35.419800 26949 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.657s user 0.010s sys 0.000s
I20240426 13:19:36.158506 25029 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:36.185180 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1689) took 1087 ms. Trace:
I20240426 13:19:36.233538 24949 rpcz_store.cc:276] 0426 13:19:35.097293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.072362 (+975069us) service_pool.cc:225] Handling call
0426 13:19:36.184741 (+112379us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.184894 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.184918 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.184959 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.184988 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.185033 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.185125 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.197865 26983 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.198321 24953 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:19:35.418217 27091 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:35.424367 27088 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.154026 25097 rpcz_store.cc:276] 0426 13:19:34.615889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.061301 (+1445412us) service_pool.cc:225] Handling call
0426 13:19:36.061587 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.153317 (+ 91730us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.153483 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:35.848652 27089 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:36.055588 26967 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.850s user 0.025s sys 0.037s
W20240426 13:19:36.199817 27065 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.942s user 0.006s sys 0.002s
W20240426 13:19:36.211091 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.009s sys 0.011s
I20240426 13:19:36.073561 27038 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.061029 27066 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.168584 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.215368 25618 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.008s sys 0.015s
W20240426 13:19:36.215528 26865 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.003s sys 0.021s
I20240426 13:19:36.168956 27036 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.059867 27061 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.230868 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:36.235603 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1697) took 1126 ms. Trace:
W20240426 13:19:36.251989 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.009s sys 0.011s
W20240426 13:19:36.300851 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 1.02181e+06us
W20240426 13:19:36.303504 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72754.8us
I20240426 13:19:36.304239 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:36.305879 24953 rpcz_store.cc:276] 0426 13:19:35.109480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.197787 (+1088307us) service_pool.cc:225] Handling call
0426 13:19:36.198110 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.235306 (+ 37196us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.235543 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.060484 26975 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.201488 27051 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.059976 27086 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.228746 27035 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.231669 26824 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.204097 27085 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.241989 26839 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:36.309499 24951 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:36.309828 24995 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.310035 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1703) took 1005 ms. Trace:
I20240426 13:19:36.547811 24949 rpcz_store.cc:276] 0426 13:19:35.304953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.253172 (+948219us) service_pool.cc:225] Handling call
0426 13:19:36.309420 (+ 56248us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.309661 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.309689 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.309742 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.309777 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.309845 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.309976 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1937,"raft.queue_time_us":109,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":14558}
W20240426 13:19:36.253688 25618 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s user 0.008s sys 0.015s
I20240426 13:19:36.328989 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.316252 26902 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:36.295958 26865 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s user 0.009s sys 0.026s
I20240426 13:19:36.414971 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.416005 27051 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.417570 27086 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.418565 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.418862 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.014s sys 0.085s
W20240426 13:19:36.585656 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.918s user 0.014s sys 0.085s
I20240426 13:19:36.413208 26824 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:36.420321 27099 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
W20240426 13:19:36.421087 27100 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240426 13:19:36.423743 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:36.424587 27104 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20240426 13:19:36.425393 26983 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:36.427173 25621 raft_consensus.cc:2354] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:19:36.272291 27089 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.253994 27091 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.326050 26993 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:36.271749 27065 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.302289 26965 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.133s
I20240426 13:19:36.432430 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:36.253886 27088 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.440493 27013 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.440912 27066 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.478425 24948 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:36.547690 24995 leader_election.cc:401] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:36.630605 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 320895us
I20240426 13:19:36.631376 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.295903 27038 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.661561 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.014s sys 0.015s
I20240426 13:19:36.412925 26977 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.595196 24951 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:36.609331 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 279) took 1284 ms. Trace:
I20240426 13:19:36.609603 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.547623 26976 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.574102 27031 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.667058 24949 rpcz_store.cc:276] 0426 13:19:35.324559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.574410 (+1249851us) service_pool.cc:225] Handling call
0426 13:19:36.608788 (+ 34378us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.608962 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.608996 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.609042 (+ 46us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.609099 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.609157 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.609273 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:36.616858 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.010s sys 0.011s
W20240426 13:19:36.620208 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 590136us
W20240426 13:19:36.627364 25620 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.034s sys 0.017s
W20240426 13:19:36.680219 25620 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s user 0.034s sys 0.017s
I20240426 13:19:36.613315 26861 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:36.629233 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1289) took 2855 ms. Trace:
I20240426 13:19:36.663522 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.681918 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.682835 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.683045 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:36.419577 26978 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:36.684731 26982 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:36.684803 26978 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.685382 26982 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:36.686245 26982 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.686914 26978 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.687436 26978 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:36.683895 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.688766 26839 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:36.667596 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46627.1us
I20240426 13:19:36.681195 25100 rpcz_store.cc:276] 0426 13:19:33.765205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.771339 (+ 6134us) service_pool.cc:225] Handling call
0426 13:19:33.836819 (+ 65480us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:33.844223 (+ 7404us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.844250 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:33.844293 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:33.844333 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.620740 (+2776407us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b54003ef8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:36.620779 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.620897 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":896,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":109,"spinlock_wait_cycles":13819264}
I20240426 13:19:36.691263 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:36.692090 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1314) took 2133 ms (client timeout 2137 ms). Trace:
W20240426 13:19:36.692394 25031 rpcz_store.cc:269] 0426 13:19:34.559039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.629004 (+ 69965us) service_pool.cc:225] Handling call
0426 13:19:36.691120 (+2062116us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:36.692020 (+ 900us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16325504}
I20240426 13:19:36.695515 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1410) took 1365 ms. Trace:
I20240426 13:19:36.698159 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1701) took 1428 ms. Trace:
I20240426 13:19:36.621492 26993 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:36.698505 24952 rpcz_store.cc:276] 0426 13:19:35.269721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.310574 (+ 40853us) service_pool.cc:225] Handling call
0426 13:19:36.698011 (+1387437us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b540038ead4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:36.698109 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1949331712}
W20240426 13:19:36.664569 27105 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.334s user 0.001s sys 0.000s
I20240426 13:19:36.684254 27066 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:36.695919 24954 rpcz_store.cc:276] 0426 13:19:35.329846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.694819 (+1364973us) service_pool.cc:225] Handling call
0426 13:19:36.695094 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.695242 (+ 148us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.695266 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.695304 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.695331 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.695376 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.695467 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.619985 27061 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:36.702880 24952 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:36.720887 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.743175 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.743791 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.750771 25098 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.707871 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.662499 27086 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.704212 25023 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:36.718420 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:36.665078 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.015s sys 0.015s
I20240426 13:19:36.757270 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.704002 26993 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.748834 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:36.749678 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47040.3us
I20240426 13:19:36.763738 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.668561 26902 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:36.752971 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.753173 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.753861 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1414) took 1360 ms. Trace:
I20240426 13:19:36.768059 24954 rpcz_store.cc:276] 0426 13:19:35.393568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.742515 (+1348947us) service_pool.cc:225] Handling call
0426 13:19:36.753377 (+ 10862us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.753529 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.753554 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.753607 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:36.753636 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.753694 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.753813 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.769028 27061 raft_consensus.cc:979] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:36.769543 27061 raft_consensus.cc:1067] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:36.769464 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.754170 25102 raft_consensus.cc:2379] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:36.754894 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.772015 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:36.714111 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.773069 25027 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:36.745311 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1411) took 1415 ms. Trace:
I20240426 13:19:36.776222 24952 rpcz_store.cc:276] 0426 13:19:35.330234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.702097 (+1371863us) service_pool.cc:225] Handling call
0426 13:19:36.702457 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.745035 (+ 42578us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.745253 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.742231 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 327) took 3175 ms. Trace:
I20240426 13:19:36.779903 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.779196 25105 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:36.753178 25097 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:36.772480 26935 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.777549 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.630956 26987 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20240426 13:19:36.786415 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.791716 27066 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:36.792526 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.698632 26977 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.793731 24952 raft_consensus.cc:2396] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240426 13:19:36.788050 25026 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.795714 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:36.745251 26839 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20240426 13:19:36.796686 27066 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.797888 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:36.798816 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.800282 25026 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:36.805495 24952 raft_consensus.cc:2354] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
I20240426 13:19:36.806625 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:36.806696 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.807363 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.810730 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1415) took 1405 ms. Trace:
I20240426 13:19:36.811187 24952 rpcz_store.cc:276] 0426 13:19:35.405266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.809198 (+1403932us) service_pool.cc:225] Handling call
0426 13:19:36.809551 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.809758 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.809795 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.810494 (+ 699us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.810527 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.810578 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.810670 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.814061 24952 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:36.814620 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1417) took 1341 ms. Trace:
I20240426 13:19:36.814983 24952 rpcz_store.cc:276] 0426 13:19:35.473312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.812289 (+1338977us) service_pool.cc:225] Handling call
0426 13:19:36.812652 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.814363 (+ 1711us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.814551 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:36.818254 25100 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:36.779719 27111 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:36.784384 26950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.000s sys 0.023s
I20240426 13:19:36.837868 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:36.794140 26902 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:36.702329 27031 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:36.839035 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 384035us
I20240426 13:19:36.802189 26975 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:36.817011 24921 connection.cc:232] RPC call timeout handler was delayed by -1.147353270999929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
I20240426 13:19:36.799042 27113 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.851485 25100 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:36.853802 25031 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:36.870189 25031 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:36.870632 25031 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:36.805096 26949 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:36.841096 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 416246us
I20240426 13:19:36.817179 27036 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:36.862633 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 532475us
I20240426 13:19:36.865242 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.839133 27112 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:36.875115 25098 raft_consensus.cc:595] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:36.877003 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 192917us
I20240426 13:19:36.839814 26978 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:36.856061 26902 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:36.842197 26950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s user 0.000s sys 0.023s
I20240426 13:19:36.884964 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.880447 25098 raft_consensus.cc:603] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:36.887799 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.888418 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.894291 25102 raft_consensus.cc:2354] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
I20240426 13:19:36.893289 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:36.895794 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.861876 27035 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.926415 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.909869 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.928395 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:36.898924 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.845811 27111 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.878069 27113 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:36.935175 25069 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.044605 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1770) took 1707 ms. Trace:
I20240426 13:19:37.129886 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1432) took 2613 ms. Trace:
W20240426 13:19:37.174762 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 239703us
I20240426 13:19:36.939993 27065 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.188009 26773 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.188303 26857 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.028213 26799 ts_tablet_manager.cc:1398] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent starting tablet: real 1.978s user 0.018s sys 0.032s
I20240426 13:19:37.174700 25099 rpcz_store.cc:276] 0426 13:19:34.516186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.520893 (+ 4707us) service_pool.cc:225] Handling call
0426 13:19:34.521153 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.555169 (+ 34016us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:34.555197 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:34.555234 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:34.555261 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.129580 (+2574319us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.129643 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.129812 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77845760}
I20240426 13:19:37.176631 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:36.978428 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:36.974815 25027 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.174362 25029 rpcz_store.cc:276] 0426 13:19:35.337590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.152050 (+814460us) service_pool.cc:225] Handling call
0426 13:19:36.152323 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.236728 (+ 84405us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.032374 (+795646us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.044536 (+ 12162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13704448}
I20240426 13:19:37.196390 25022 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:37.173280 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 308150us
I20240426 13:19:37.197623 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.198055 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.889300 27085 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.180281 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 246896us
W20240426 13:19:37.133129 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1704) took 1827 ms (client timeout 2368 ms). Trace:
W20240426 13:19:37.201211 24951 rpcz_store.cc:269] 0426 13:19:35.305273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.327001 (+ 21728us) service_pool.cc:225] Handling call
0426 13:19:37.132984 (+1805983us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b540030c5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.133076 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2253267712}
I20240426 13:19:36.880402 26949 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.901062 27036 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:37.024406 27031 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.005756 26744 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:37.211788 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.020212 27091 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.047942 27054 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.663s user 0.018s sys 0.052s
I20240426 13:19:37.198654 25029 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.219026 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.219911 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:37.054740 27118 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
W20240426 13:19:37.054069 27124 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240426 13:19:36.896916 27089 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.204207 27023 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.204900 26949 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.203320 26656 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:37.005424 27038 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:36.958014 26982 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.048481 27068 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:37.179774 27016 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.213786 26713 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.208388 26952 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.194048 26824 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.580s user 0.008s sys 0.001s
I20240426 13:19:37.055158 26977 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.175736 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:37.253698 27120 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.208s user 0.000s sys 0.002s
I20240426 13:19:37.201656 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:37.262356 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 60815.1us
I20240426 13:19:37.263113 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.199110 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.208845 26945 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20240426 13:19:37.217475 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1696) took 2108 ms. Trace:
W20240426 13:19:37.197508 26799 ts_tablet_manager.cc:1430] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Tablet startup took 2163ms
I20240426 13:19:36.997896 27088 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:36.963431 27102 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:36.879437 27116 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.272281 25616 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:37.271976 27116 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.203171 27013 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.000s
I20240426 13:19:37.275219 26839 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.223158 26744 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.275925 26839 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.276865 27102 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:36.899456 26912 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.307s
I20240426 13:19:37.227000 25589 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:37.279635 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52674.2us
I20240426 13:19:37.229329 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1444) took 2710 ms. Trace:
W20240426 13:19:37.231935 27119 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
I20240426 13:19:37.209331 26859 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20240426 13:19:37.281764 25104 rpcz_store.cc:276] 0426 13:19:34.519233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.607133 (+ 87900us) service_pool.cc:225] Handling call
0426 13:19:34.607409 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:34.864375 (+256966us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.229095 (+2364720us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.229266 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577283456}
I20240426 13:19:37.234086 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1766) took 1909 ms. Trace:
I20240426 13:19:37.283167 25025 rpcz_store.cc:276] 0426 13:19:35.324801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.062435 (+737634us) service_pool.cc:225] Handling call
0426 13:19:36.180716 (+118281us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.414263 (+233547us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.414306 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.414359 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.414398 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.233850 (+819452us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.233894 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.234020 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536591744}
I20240426 13:19:37.238317 27107 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:19:37.242096 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1412) took 1904 ms (client timeout 1953 ms). Trace:
I20240426 13:19:37.245050 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1678) took 3705 ms. Trace:
I20240426 13:19:37.226805 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.286799 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.285553 24955 rpcz_store.cc:276] 0426 13:19:33.539949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.775670 (+235721us) service_pool.cc:225] Handling call
0426 13:19:33.775929 (+ 259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:33.811202 (+ 35273us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.811245 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:33.819449 (+ 8204us) log.cc:842] Serialized 26 byte log entry
0426 13:19:34.273938 (+454489us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:34.386350 (+112412us) log.cc:842] Serialized 14 byte log entry
0426 13:19:34.386550 (+ 200us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:34.386650 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.047404 (+2660754us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:37.244706 (+197302us) raft_consensus.cc:1671] finished
0426 13:19:37.244842 (+ 136us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.244982 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15062,"prepare.queue_time_us":24961,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":3772,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":129527,"spinlock_wait_cycles":3638016,"thread_start_us":516995,"threads_started":2,"wal-append.queue_time_us":469701}
I20240426 13:19:37.247270 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.040668 25031 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:37.289191 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91206.8us
I20240426 13:19:37.289711 25031 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:37.289911 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.289816 26902 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.259022 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.127960 27136 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.299867 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.292002 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:36.889837 27112 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.264549 25027 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:37.258175 27068 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:37.269091 24995 connection.cc:232] RPC call timeout handler was delayed by -1.281306637000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20240426 13:19:37.264547 24947 rpcz_store.cc:276] 0426 13:19:35.109203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.072644 (+963441us) service_pool.cc:225] Handling call
0426 13:19:36.073127 (+ 483us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.679712 (+606585us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.682070 (+ 2358us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:36.682536 (+ 466us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.869180 (+186644us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.869236 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.869327 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.217223 (+347896us) raft_consensus.cc:1671] finished
0426 13:19:37.217249 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.217412 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114614,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":791499,"threads_started":2,"wal-append.queue_time_us":346172}
I20240426 13:19:37.270334 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.271302 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.277280 26744 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.261895 26824 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.290319 27020 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:37.291225 25099 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:37.291533 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 115871us
I20240426 13:19:37.262969 26983 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.257988 26982 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:37.284801 24948 rpcz_store.cc:269] 0426 13:19:35.337353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.362476 (+ 25123us) service_pool.cc:225] Handling call
0426 13:19:37.241938 (+1879462us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.242032 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":490422144}
I20240426 13:19:37.223026 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.294113 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.290885 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1698) took 2181 ms. Trace:
I20240426 13:19:37.323462 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.302937 25022 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:37.303566 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:37.308207 25029 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.292995 27107 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.301532 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.280129 26747 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.339668 25025 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:36.966812 27035 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.310973 25027 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.318548 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.268733 27038 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.319650 25623 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.324744 25103 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.371992 26839 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.372124 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.297217 27082 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:37.373929 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 78832.1us
I20240426 13:19:37.381572 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:37.381723 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63242.4us
I20240426 13:19:37.321910 24956 rpcz_store.cc:276] 0426 13:19:35.113229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.226961 (+1113732us) service_pool.cc:225] Handling call
0426 13:19:36.227415 (+ 454us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:36.233971 (+ 6556us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.234014 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:36.241602 (+ 7588us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.745715 (+504113us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:36.745750 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.745818 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.231365 (+485547us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:37.290626 (+ 59261us) raft_consensus.cc:1671] finished
0426 13:19:37.290643 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.290829 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24727,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":41652,"spinlock_wait_cycles":18320128,"thread_start_us":417203,"threads_started":1,"wal-append.queue_time_us":669418}
I20240426 13:19:37.387336 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.312820 26969 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:37.393986 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1295) took 3277 ms (client timeout 1884 ms). Trace:
I20240426 13:19:37.308701 27013 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.308833 27088 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.262408 26713 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.296770 27136 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:37.312767 26824 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.446843 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.388936 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.375007 27066 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.392496 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.322721 26982 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.431417 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.436323 26977 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.463347 27066 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.463909 27066 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.464471 26902 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.459601 27102 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.385877 27082 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.466312 27082 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.469377 25104 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:19:37.448412 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 59578.4us
I20240426 13:19:37.373896 26747 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:37.394480 25101 rpcz_store.cc:269] 0426 13:19:34.107210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.206033 (+ 98823us) service_pool.cc:225] Handling call
0426 13:19:37.384803 (+3178770us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54000eb354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.384905 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429277056}
W20240426 13:19:37.444187 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52640.1us
I20240426 13:19:37.333030 27020 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:37.477062 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 84594.1us
I20240426 13:19:37.379290 27085 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.090s
I20240426 13:19:37.474153 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.310187 27068 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.472790 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.442288 26839 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.472499 25099 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.501579 26839 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.502158 26839 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.504021 26679 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:37.504526 26679 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.505389 26679 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.506542 26679 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:36.780496 25618 rpcz_store.cc:276] 0426 13:19:33.523234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.525619 (+ 2385us) service_pool.cc:225] Handling call
0426 13:19:33.537271 (+ 11652us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:33.553639 (+ 16368us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:33.556097 (+ 2458us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:33.701065 (+144968us) log.cc:842] Serialized 26 byte log entry
0426 13:19:35.376276 (+1675211us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b1800118cc0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:36.585415 (+1209139us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.605122 (+ 19707us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:36.613581 (+ 8459us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:36.616099 (+ 2518us) raft_consensus.cc:1671] finished
0426 13:19:36.617100 (+ 1001us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:36.698977 (+ 81877us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8665,"raft.queue_time_us":12524,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":105532,"spinlock_wait_cycles":65238912,"thread_start_us":1195068,"threads_started":1,"wal-append.queue_time_us":2631503}
I20240426 13:19:37.473177 27107 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.509245 27107 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.510643 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:37.511656 26839 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.512972 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.513911 27066 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.516214 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1409) took 2187 ms. Trace:
I20240426 13:19:37.516595 24949 rpcz_store.cc:276] 0426 13:19:35.328719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.671982 (+1343263us) service_pool.cc:225] Handling call
0426 13:19:36.672433 (+ 451us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.743217 (+ 70784us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.743251 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.743293 (+ 42us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:36.743324 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.515956 (+772632us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.516017 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.516155 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162317568}
I20240426 13:19:37.482522 27020 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.485625 26721 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:37.529616 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:37.469175 26983 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:37.538275 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1769) took 2212 ms. Trace:
I20240426 13:19:37.458753 27035 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.489085 26987 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:37.477099 26747 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.489969 24853 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 173) took 6736 ms. Trace:
I20240426 13:19:37.491019 27082 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.492165 25097 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.492223 26978 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.496847 24851 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.499181 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1447) took 2958 ms. Trace:
I20240426 13:19:37.468171 27096 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.514135 26839 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.485174 26713 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:37.520799 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48076.3us
W20240426 13:19:37.601756 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48046us
I20240426 13:19:37.486240 27112 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.478293 27051 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20240426 13:19:37.477502 27088 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:37.532476 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 241) took 3645 ms (client timeout 2754 ms). Trace:
W20240426 13:19:37.630228 25024 rpcz_store.cc:269] 0426 13:19:33.886835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.887933 (+ 1098us) service_pool.cc:225] Handling call
0426 13:19:37.532307 (+3644374us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54001e3554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.532409 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":947230592}
I20240426 13:19:37.540129 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 480) took 2266 ms. Trace:
I20240426 13:19:37.540645 24996 leader_election.cc:398] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5.
W20240426 13:19:37.631652 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 91137.5us
I20240426 13:19:37.541411 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 278) took 2217 ms. Trace:
I20240426 13:19:37.550189 27068 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.538652 25030 rpcz_store.cc:276] 0426 13:19:35.325380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.150920 (+825540us) service_pool.cc:225] Handling call
0426 13:19:36.151166 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.194355 (+ 43189us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.194385 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.194421 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.194450 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.538091 (+1343641us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400056ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.538131 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.538227 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98935552}
I20240426 13:19:37.546599 27141 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:37.638420 27141 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.537609 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1762) took 2213 ms. Trace:
I20240426 13:19:37.591857 27115 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.442237 26969 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.555334 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.645700 25101 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:37.588238 25096 rpcz_store.cc:276] 0426 13:19:34.540310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.315001 (+774691us) service_pool.cc:225] Handling call
0426 13:19:35.315292 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.332180 (+1016888us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.332216 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.332269 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:36.332312 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.498974 (+1166662us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.499014 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.499128 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2338310400}
I20240426 13:19:37.647107 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:37.562491 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88394.8us
I20240426 13:19:37.477191 27116 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.478646 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.539319 27061 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.604668 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.601365 26964 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.607715 25099 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.608701 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.609150 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.656003 24954 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:37.609977 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.593132 24851 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.540436 26935 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.612253 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.616578 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:37.659665 24950 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.660143 24955 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:37.619184 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.620679 27013 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.621101 26952 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.621615 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.622196 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.669405 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.623718 27082 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.624485 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.673899 25624 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.624644 24952 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:37.601948 27035 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.625736 26839 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.678937 26839 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.680745 26839 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.587672 26978 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.539113 26656 consensus_queue.cc:1159] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:37.683715 26993 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.903s user 0.009s sys 0.002s
I20240426 13:19:37.684305 26993 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.686756 26993 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.645305 27115 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.601073 27096 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.549870 27111 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.632899 24953 rpcz_store.cc:276] 0426 13:19:35.324354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:36.306830 (+982476us) service_pool.cc:225] Handling call
0426 13:19:36.327385 (+ 20555us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.665664 (+338279us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.665709 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.665761 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:36.665798 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.541134 (+875336us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.541195 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.541337 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777471360}
W20240426 13:19:37.268433 26799 ts_tablet_manager.cc:1432] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Trace:
0426 13:19:26.765217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:26.765367 (+ 150us) service_pool.cc:225] Handling call
0426 13:19:27.070272 (+304905us) ts_tablet_manager.cc:903] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Initiating tablet copy from peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
0426 13:19:29.296472 (+2226200us) tablet_copy_client.cc:353] Tablet copy session begun
0426 13:19:30.014282 (+717810us) tablet_metadata.cc:641] Metadata flushed
0426 13:19:30.014676 (+ 394us) tablet_copy_client.cc:482] Wrote new tablet metadata
0426 13:19:30.181366 (+166690us) tablet_copy_client.cc:403] Wrote new consensus metadata
0426 13:19:30.181821 (+ 455us) tablet_replica.cc:205] Creating consensus instance
0426 13:19:30.182267 (+ 446us) ts_tablet_manager.cc:958] Replica registered.
0426 13:19:30.182364 (+ 97us) inbound_call.cc:173] Queueing success response
0426 13:19:32.446086 (+2263722us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0426 13:19:35.057352 (+2611266us) ts_tablet_manager.cc:1399] Starting tablet replica
0426 13:19:35.104464 (+ 47112us) tablet_replica.cc:259] Starting instrumentation
0426 13:19:35.105013 (+ 549us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":25453,"mutex_wait_us":126335,"raft.queue_time_us":149965,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":7022,"tablet-copy.queue_time_us":173608,"tablet-download-pool-.queue_time_us":1911,"tablet-download-pool-.run_cpu_time_us":1411,"tablet-download-pool-.run_wall_time_us":1148283,"thread_start_us":1426950,"threads_started":6}
I20240426 13:19:37.632494 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.555270 24853 rpcz_store.cc:276] 0426 13:19:30.837365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:30.837634 (+ 269us) service_pool.cc:225] Handling call
0426 13:19:37.489903 (+6652269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:32.050702 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:32.220420 (+169718us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:32.220479 (+ 59us) write_op.cc:427] Acquired schema lock
0426 13:19:32.220497 (+ 18us) tablet.cc:660] Decoding operations
0426 13:19:32.221589 (+ 1092us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:32.221722 (+ 133us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:32.221757 (+ 35us) tablet.cc:683] Acquiring locks for 9 operations
0426 13:19:32.222642 (+ 885us) tablet.cc:699] Row locks acquired
0426 13:19:32.222677 (+ 35us) write_op.cc:260] PREPARE: finished
0426 13:19:32.222828 (+ 151us) write_op.cc:270] Start()
0426 13:19:32.222967 (+ 139us) write_op.cc:275] Timestamp: P: 1714137572222770 usec, L: 0
0426 13:19:32.222997 (+ 30us) op_driver.cc:331] REPLICATION: starting
0426 13:19:32.223643 (+ 646us) log.cc:842] Serialized 3547 byte log entry
0426 13:19:32.910145 (+686502us) op_driver.cc:444] REPLICATION: finished
0426 13:19:33.306451 (+396306us) write_op.cc:300] APPLY: starting
0426 13:19:33.341234 (+ 34783us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:33.353816 (+ 12582us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:33.359507 (+ 5691us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:33.417538 (+ 58031us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:33.417826 (+ 288us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0426 13:19:33.417875 (+ 49us) write_op.cc:311] APPLY: finished
0426 13:19:33.418217 (+ 342us) log.cc:842] Serialized 71 byte log entry
0426 13:19:33.418403 (+ 186us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:33.418733 (+ 330us) write_op.cc:446] Released schema lock
0426 13:19:33.418807 (+ 74us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:33.419068 (+ 261us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:33.419095 (+ 27us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":788,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":285566,"apply.run_cpu_time_us":4635,"apply.run_wall_time_us":139635,"num_ops":9,"prepare.queue_time_us":446026,"prepare.run_cpu_time_us":17637,"prepare.run_wall_time_us":442319,"raft.queue_time_us":130834,"raft.run_cpu_time_us":24100,"raft.run_wall_time_us":291145,"replication_time_us":612415,"thread_start_us":727136,"threads_started":4,"wal-append.queue_time_us":342193,"wal-append.run_cpu_time_us":10356,"wal-append.run_wall_time_us":1955326}]]}
I20240426 13:19:37.639158 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.639775 25028 rpcz_store.cc:276] 0426 13:19:35.323903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.331210 (+ 7307us) service_pool.cc:225] Handling call
0426 13:19:35.485659 (+154449us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:36.418467 (+932808us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:36.418495 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:36.418531 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:36.418557 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.537396 (+1118839us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54001e3054. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:37.537439 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.537553 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2144953088}
I20240426 13:19:37.642099 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.639111 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.707651 24950 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.601512 26757 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.651325 25097 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:37.712198 25097 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:37.470999 26859 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.651434 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 172872us
W20240426 13:19:37.628612 27143 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240426 13:19:37.668593 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.652283 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.631322 25620 rpcz_store.cc:276] 0426 13:19:35.274076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:35.275770 (+ 1694us) service_pool.cc:225] Handling call
0426 13:19:35.290593 (+ 14823us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:35.291023 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:35.291052 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:35.291593 (+ 541us) log.cc:842] Serialized 26 byte log entry
0426 13:19:36.070786 (+779193us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:36.680649 (+609863us) log.cc:842] Serialized 14 byte log entry
0426 13:19:37.539685 (+859036us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b540053e420. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902ab143 00007fda9031809f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c
0426 13:19:37.539747 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:37.539834 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:37.539876 (+ 42us) raft_consensus.cc:1671] finished
0426 13:19:37.539950 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:37.540077 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":729277,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":32383,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"spinlock_wait_cycles":48510848,"thread_start_us":1388189,"threads_started":2,"wal-append.queue_time_us":866746}
W20240426 13:19:37.653620 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 217752us
I20240426 13:19:37.561703 27084 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:37.608085 27113 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.657701 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.657738 24954 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:37.662257 24851 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:19:37.723173 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 65549.4us
I20240426 13:19:37.724261 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:37.678112 27035 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.702903 27115 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.560057 26945 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.705400 27066 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:37.661715 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 334) took 3087 ms (client timeout 1581 ms). Trace:
W20240426 13:19:37.707878 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75471.2us
I20240426 13:19:37.738313 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.483629 27136 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.708879 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66864.1us
I20240426 13:19:37.676406 26952 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.744202 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.714998 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.745151 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:37.745555 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:37.745744 25619 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.746698 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.639174 27141 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.747846 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.719691 24951 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:37.724107 24851 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:19:37.749770 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.750396 24851 catalog_manager.cc:5484] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:19:37.751564 26861 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:37.660161 27112 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.750393 26413 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 13:19:37.756940 26413 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:37.757279 26413 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.757711 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:37.758126 26413 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.759479 26413 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:37.665720 26935 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:37.731408 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.953s (SENT)
I20240426 13:19:37.711679 26993 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.717213 26839 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.728345 27029 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.761536 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:37.744578 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:37.730674 26757 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:37.768352 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:37.752421 27141 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.788448 26757 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:37.788926 26757 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.672737 27013 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.753331 27066 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.753734 27111 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.713222 25097 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:19:37.729969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127242us
I20240426 13:19:37.757293 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.736904 26945 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:37.736752 25621 rpcz_store.cc:269] 0426 13:19:34.564480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:34.564682 (+ 202us) service_pool.cc:225] Handling call
0426 13:19:37.636074 (+3071392us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54006bf8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:37.651458 (+ 15384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3902220160}
I20240426 13:19:37.746961 27085 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.728582 27084 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.761914 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.769727 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:37.752936 27035 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:37.791857 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62122.2us
I20240426 13:19:37.802246 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.756134 24851 catalog_manager.cc:5484] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 3, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 3 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.793957 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.850466 25030 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.850687 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.851253 25026 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.852084 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.852944 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.762625 24950 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:37.754277 27113 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.801923 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.856211 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.856832 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:37.842486 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80649us
I20240426 13:19:37.811223 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.860297 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.835016 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.812389 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:37.794658 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.787117 24952 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:37.794658 25031 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240426 13:19:37.866830 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.867550 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:37.796876 25029 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.800349 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.795047 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.795743 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.869661 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.846065 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.795751 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.846724 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.846288 25099 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:37.871055 25101 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:37.850128 24851 catalog_manager.cc:5484] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 4, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.854887 26965 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:37.872402 26965 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:37.872602 26861 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.872783 26965 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:37.873034 26861 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:37.873917 26861 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.875066 26861 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:37.876096 27114 consensus_queue.cc:1159] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
W20240426 13:19:37.876103 26996 consensus_queue.cc:1159] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 4 status { last_received { term: 4 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 4 index: 6 } }
I20240426 13:19:37.878139 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.786578 26969 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:37.772881 26857 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.576s user 0.004s sys 0.004s
I20240426 13:19:37.800791 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.856707 25022 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.885174 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.882930 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.885193 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.888341 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.898254 25103 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:37.889307 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.780954 27136 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.898236 25102 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:37.901372 24948 raft_consensus.cc:2396] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:37.901994 25097 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:37.793994 27066 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.761823 26894 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.769560 27039 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:37.895282 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.794370 27102 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:37.813267 27112 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:37.925473 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:37.904597 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 427844us
I20240426 13:19:37.783622 27096 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.913688 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.928936 25100 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:37.915629 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65078.3us
I20240426 13:19:37.783986 27029 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:37.778425 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:37.917624 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 59212.6us
I20240426 13:19:37.952302 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.885546 24851 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: config changed from index -1 to 2, NON_VOTER 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) added. New cstate: current_term: 1 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:37.957083 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:37.959750 25069 leader_election.cc:398] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
I20240426 13:19:37.923347 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.924108 25022 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:37.761766 27036 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.152s
I20240426 13:19:37.907347 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.896317 27035 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:37.771819 26859 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:37.896324 27111 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.904953 26656 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:37.926108 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:37.939574 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:37.915642 26962 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.913602 26839 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:37.879874 26982 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.919687 27088 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20240426 13:19:37.904348 27115 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.961632 27147 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:37.923197 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:37.913801 27107 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:37.954779 24851 catalog_manager.cc:5484] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:38.036445 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.037781 25101 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:37.985550 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:37.998481 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.972214 26413 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20240426 13:19:38.039840 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:37.987000 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:38.032878 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109652us
I20240426 13:19:37.973958 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.041492 25023 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.041657 25031 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
I20240426 13:19:38.039369 25097 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:38.033006 25069 leader_election.cc:401] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240426 13:19:37.922204 27066 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.045531 25096 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:37.921590 27039 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:38.057093 25103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:38.057878 25103 raft_consensus.cc:595] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.058310 25103 raft_consensus.cc:603] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:38.040472 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:38.060827 25027 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.056155 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.062434 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.062815 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.063920 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.064945 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.065104 25025 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:37.948403 27029 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.065876 25102 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:37.988014 25041 tablet_copy_service.cc:342] P 46106ff992984d33bb4a4d8f30162639: Request end of tablet copy session 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309 received from {username='slave'} at 127.0.0.1:47174
I20240426 13:19:38.069017 24996 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:38.069716 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 110207us
I20240426 13:19:38.070010 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:37.962567 27013 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:37.964519 27112 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.075467 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.044507 26656 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:37.985709 26935 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:37.968081 26945 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:37.968022 26721 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.074867 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.001917 27084 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.033340 26839 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:38.077684 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 311370us
I20240426 13:19:38.061655 27035 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.068652 27096 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.100142 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.100842 24952 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.100858 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.103404 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.036131 27106 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:37.986550 26952 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.046581 26894 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.046330 27107 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.084830 26413 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:38.079705 27020 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.122195 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.046221 27111 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.087312 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.097908 26721 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.002542 26757 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:37.979940 26965 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:37.981194 26747 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.115999 26413 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:37.947903 26824 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.131s
I20240426 13:19:38.075840 27150 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6.
W20240426 13:19:38.117275 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 304999us
I20240426 13:19:38.036201 26962 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.082676 26656 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.085026 27036 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:38.068801 25618 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.075764 25041 tablet_copy_service.cc:433] P 46106ff992984d33bb4a4d8f30162639: ending tablet copy session 1590f80bdade41c6a755b8501c55fe16-11fad756437243d2959f5998d6bf8309 on tablet 11fad756437243d2959f5998d6bf8309 with peer 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:38.132159 26952 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.125892 25098 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.154853 24952 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.128917 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.133164 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.099184 26839 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.081022 27147 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240426 13:19:38.136929 26413 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.162916 26413 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.118402 27035 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.127497 27020 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.123608 27107 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.115113 26773 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.082607 27068 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 13:19:38.086032 27051 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20240426 13:19:38.133666 27061 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:38.148403 25619 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.125864 26977 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.151423 25026 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.149435 25588 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.184098 27096 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.184708 27096 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.185005 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.186614 27096 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.186750 27113 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.187129 27096 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:38.130292 26721 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.098817 27013 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.156724 27036 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.144184 25615 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:38.157745 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.134168 27088 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.188832 27036 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:38.190112 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.159312 25030 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.193084 27084 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:38.193677 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.194449 27084 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.194849 27084 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.195171 27084 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.197296 27111 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.198580 27088 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.175979 27112 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.201701 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.136379 26894 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.202859 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.203871 26757 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.204391 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:38.205546 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 477) took 4654 ms (client timeout 1645 ms). Trace:
I20240426 13:19:38.205499 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.206430 25617 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.159744 25096 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:38.120046 27106 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.191831 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:38.160825 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 304078us
I20240426 13:19:38.209251 27106 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.152796 27114 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.209815 27114 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.210606 27114 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.211557 27114 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.211432 27085 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:38.203805 27051 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.135681 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.215837 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.212215 27111 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.217062 27111 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.217448 27111 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.202883 26894 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.217746 26747 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.212105 26952 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.212433 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.226867 25619 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:38.212798 26912 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:38.185088 26973 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.221336 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:38.206041 25615 rpcz_store.cc:269] 0426 13:19:33.551265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.658303 (+107038us) service_pool.cc:225] Handling call
0426 13:19:38.205382 (+4547079us) spinlock_profiling.cc:243] Waited 4.33 s on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:38.205464 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9954013568}
I20240426 13:19:38.203946 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.223012 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.242242 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.241055 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.242725 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.243690 24948 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.248309 24920 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.153832 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.247126 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:38.256413 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 144556us
I20240426 13:19:38.177098 27020 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.166071 27016 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:38.262082 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.262970 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.265271 24920 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.266769 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.187976 27096 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:38.181417 26773 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.272161 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.201735 26962 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.260497 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:38.273722 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.279011 26961 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:38.279165 27152 log.cc:924] Time spent T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20240426 13:19:38.279510 26961 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.272383 27096 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.279968 26961 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.281006 27096 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.282390 27096 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:38.283217 26978 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.002s
I20240426 13:19:38.284317 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:38.284951 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 146730us
I20240426 13:19:38.287243 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.245944 26973 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.275182 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240426 13:19:38.257625 27088 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.264593 27113 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.297514 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.257369 27061 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.288151 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.300969 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.272943 27013 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.267915 27068 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.292791 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:19:38.305881 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.273309 26773 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.310333 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.313536 25096 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:38.289227 27016 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:38.288722 27115 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.322994 26413 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.323448 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.291168 27038 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:38.320772 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.296368 26973 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.328359 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:38.307281 27085 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:38.297255 27106 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.332978 26413 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.308190 26969 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.283615 26978 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.293306 27112 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.320363 27113 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.361479 27113 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:38.362323 27113 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.328063 26713 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.365382 26952 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.366586 27061 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.369848 27039 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.371206 27039 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.320484 27088 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.315690 26982 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.353832 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:38.388633 27035 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.323911 27016 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.389338 27016 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.356055 26969 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.390170 26969 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.327936 26985 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:38.392244 26985 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.392632 26985 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.394919 25023 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
W20240426 13:19:38.388525 27159 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240426 13:19:38.311828 27102 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240426 13:19:38.335219 27038 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.430531 26912 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.430697 27038 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:38.430931 26912 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.431743 26912 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.433112 26912 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.348958 27115 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:19:38.427651 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 73950.8us
I20240426 13:19:38.429056 27085 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.439963 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.440191 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.441556 25096 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.354418 26965 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.460405 26969 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.461428 26945 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.461918 26945 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.462631 26978 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.463043 26945 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.443817 27168 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20240426 13:19:38.439435 26747 consensus_queue.cc:1159] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 3 index: 3 } }
I20240426 13:19:38.466022 26977 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:38.446247 26985 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.448920 27029 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.466434 26977 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.467319 26977 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.467885 27029 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.454778 26656 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.471792 26656 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.472182 27085 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.472334 26965 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:38.472896 26965 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.473315 27085 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.460405 26773 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.427945 27160 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240426 13:19:38.468240 25620 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.474886 27107 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.468808 27088 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.481081 25031 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.481125 27088 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.483038 27088 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.460137 27020 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.485782 27016 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:38.458295 27136 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.538s user 0.008s sys 0.000s
I20240426 13:19:38.473325 26965 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:38.515982 27016 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.447533 26973 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.518530 25621 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.457613 26721 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.520805 25104 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.520954 27084 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.352360 26757 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20240426 13:19:38.469736 26985 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.489015 26961 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.443152 27111 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.538372 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.538990 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.539346 27111 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.499131 27066 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.513759 27038 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.536198 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.537330 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:38.550916 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 284229us
I20240426 13:19:38.551648 24920 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.552358 27111 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.552695 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.543185 25096 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:38.553409 27111 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:38.541913 26977 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.553942 27111 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.554977 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:38.560751 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 236390us
I20240426 13:19:38.561141 25624 raft_consensus.cc:2379] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.554461 27111 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.562970 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.562918 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:38.565500 25031 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:38.541815 26945 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:38.558521 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.533684 26969 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.571615 24947 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.541497 26973 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.572840 25023 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.574997 26413 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.602862 26413 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.575496 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:38.539350 26961 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.572345 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:38.604751 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.605506 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.606112 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:38.593444 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 270060us
I20240426 13:19:38.595454 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.598341 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.554847 26978 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20240426 13:19:38.561758 27096 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:38.603410 26413 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.541509 26773 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:38.599699 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.617923 25104 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.530068 26965 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.568886 27020 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:38.628924 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56661.9us
I20240426 13:19:38.568565 26987 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:38.601249 27016 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.607048 27084 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.632961 26993 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:38.601732 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.569553 27162 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20240426 13:19:38.645427 27108 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.654476 26413 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.657372 24955 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.641157 26993 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:38.653591 27108 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.658766 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.683845 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.649967 27036 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:38.628667 27096 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:38.661446 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 374261us
I20240426 13:19:38.669157 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.670569 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.691900 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.693092 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.694864 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.668982 27112 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.654171 26773 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.674713 25104 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.641515 27016 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.683579 26993 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.686177 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.697959 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.700711 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.679965 26413 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.716192 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.716341 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.717444 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:38.706897 24950 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:38.707861 25589 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:38.687986 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:38.722047 27170 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20240426 13:19:38.724431 26413 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:38.725554 26413 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:38.725893 26413 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.726727 26413 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.727753 26413 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.728562 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.729496 24952 raft_consensus.cc:2354] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:38.730193 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.730870 24952 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.731528 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.732802 24952 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.732901 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.734714 25624 raft_consensus.cc:2354] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:38.735850 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.736299 26912 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.736703 24950 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:38.732447 27091 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:38.738443 27091 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:38.738997 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.739657 24948 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.739240 27091 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:38.740361 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.701591 27162 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.741142 24951 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
W20240426 13:19:38.740805 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52937.6us
I20240426 13:19:38.750420 25101 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.705046 26773 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:38.743870 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.706310 27051 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.700647 27150 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.701653 27066 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.701400 27036 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.754168 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:38.705559 27061 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.757967 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:38.741724 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 453668us
I20240426 13:19:38.745723 26721 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:38.760784 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:38.726902 27141 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.762091 25026 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:38.724833 27146 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.722527 27163 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.781352 24952 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:38.805368 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.782299 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.806360 25624 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.795661 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.808444 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.809280 25105 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:38.810366 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.793274 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.794834 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.794912 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.811419 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.797472 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.812211 25097 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.812211 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.798854 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.798346 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.797168 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.812973 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.813552 25097 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:38.814076 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.797756 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.797809 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.799046 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.801743 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.826258 25100 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:38.812542 25031 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.796850 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.820495 27171 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:38.781414 26965 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:38.813266 25099 raft_consensus.cc:2362] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6. Re-sending same reply.
W20240426 13:19:38.823781 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63110.8us
I20240426 13:19:38.807898 27036 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.819224 24952 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:38.820331 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:38.824316 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.828390 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.828516 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.840283 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.830528 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.840926 25022 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.827653 25620 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.832518 25024 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:38.832442 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.833086 25029 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:38.840516 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.842701 27084 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:38.844020 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.846827 25097 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:38.846944 27066 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.847468 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.722262 27108 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.856123 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.857507 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.840435 25026 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.860242 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.860952 25029 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.846071 24949 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.862776 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.846120 24952 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.849478 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:38.866508 24921 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.866798 24921 leader_election.cc:401] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:38.876646 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:38.877672 24948 raft_consensus.cc:2379] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:38.878933 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 78077.3us
I20240426 13:19:38.834589 26985 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.891647 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:38.895233 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.896633 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.897799 25029 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:38.898691 25027 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:38.878449 24952 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:38.866986 27179 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:38.905004 24948 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:38.906291 24955 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.907377 24949 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:38.884477 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:38.888650 27107 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.849164 26773 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.909271 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:38.911777 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63054us
I20240426 13:19:38.912406 24952 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:19:38.914028 24951 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:38.915391 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.918399 25099 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:38.897061 27108 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:38.921542 25031 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:38.888669 27096 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:38.898823 27066 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:38.923110 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:38.923938 24948 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:38.924636 25622 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:38.923609 25069 leader_election.cc:398] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:38.929317 25069 leader_election.cc:401] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:38.910532 26656 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:38.909880 27084 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:38.916427 27088 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.910233 26985 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:38.967710 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52249.5us
I20240426 13:19:38.968731 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:38.917480 27157 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:38.969141 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45661.6us
I20240426 13:19:38.976131 25069 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:38.978518 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.904039 27136 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20240426 13:19:38.924976 27112 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:38.910828 27035 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240426 13:19:38.990202 25102 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:38.848690 27171 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:38.993213 27106 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:19:38.995620 27175 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20240426 13:19:38.996702 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:38.978055 26656 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:39.001025 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:38.919407 27162 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240426 13:19:39.004670 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:38.993352 27114 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.781s user 0.014s sys 0.000s
I20240426 13:19:39.016706 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.033514 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.035410 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.035656 25030 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:38.928107 27036 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:38.977658 26978 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:38.996484 27150 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:38.930164 27179 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.085s
I20240426 13:19:39.041507 25027 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:39.041934 25031 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 13:19:39.050055 25031 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 1
I20240426 13:19:39.050287 25031 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 13:19:39.050549 25031 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:39.013969 26679 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.006269 27016 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.059409 25030 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:39.013860 27106 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:38.997684 26757 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20240426 13:19:38.988782 27157 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.033797 27147 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.024627 26983 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:39.034343 27093 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:39.040391 25621 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
W20240426 13:19:39.093371 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 117373us
I20240426 13:19:39.105574 25102 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
W20240426 13:19:39.144591 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 287) took 1380 ms (client timeout 1500 ms). Trace:
W20240426 13:19:39.166352 24947 rpcz_store.cc:269] 0426 13:19:37.764014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:37.768141 (+ 4127us) service_pool.cc:225] Handling call
0426 13:19:39.144340 (+1376199us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54000bf754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:39.144504 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800937728}
I20240426 13:19:39.043851 27187 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.015820 27085 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240426 13:19:39.045439 26996 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.155045 25105 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:39.115233 27192 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.284s user 0.001s sys 0.000s
I20240426 13:19:39.052313 27029 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20240426 13:19:39.168881 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:19:39.121034 27191 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20240426 13:19:39.175455 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.188406 25622 raft_consensus.cc:2354] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
I20240426 13:19:39.188992 27136 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:39.158926 27196 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20240426 13:19:39.121349 26679 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.190409 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.045480 27036 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.051862 26978 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.195161 25620 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20240426 13:19:39.197517 27195 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
W20240426 13:19:39.214879 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 124083us
W20240426 13:19:39.198297 27194 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.424s user 0.005s sys 0.000s
I20240426 13:19:39.202716 25069 leader_election.cc:398] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
I20240426 13:19:39.215780 25069 leader_election.cc:401] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:39.204838 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48646us
I20240426 13:19:39.186501 27107 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:39.213953 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 291104us
I20240426 13:19:39.212572 24920 leader_election.cc:398] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5.
W20240426 13:19:39.282621 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 79293us
I20240426 13:19:39.283200 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.286429 24949 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:39.287304 24951 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.287408 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:39.260372 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57817.8us
I20240426 13:19:39.289319 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.289350 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.291541 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.293318 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.294566 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:39.260900 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 43981.1us
I20240426 13:19:39.295871 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.298317 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.299034 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.300626 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.060309 26985 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:39.211855 26839 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.029s user 0.012s sys 0.005s
I20240426 13:19:39.192880 26861 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.208896 26996 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.262658 27162 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.209707 27136 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240426 13:19:39.266319 27111 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.692s user 0.008s sys 0.000s
I20240426 13:19:39.319243 27111 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.320415 24954 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:39.320824 26945 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.728s user 0.004s sys 0.004s
I20240426 13:19:39.186998 27106 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.321427 25104 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:39.158031 26656 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.322503 26965 consensus_queue.cc:1159] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:39.324904 27111 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.260601 27146 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3.
W20240426 13:19:39.273041 27190 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20240426 13:19:39.325479 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:39.208626 27147 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.327093 27147 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.328969 25103 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:39.273725 26978 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.275571 27171 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.276104 24948 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240426 13:19:39.277062 27157 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.277400 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.277447 27203 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240426 13:19:39.278621 26679 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.199860 27187 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20240426 13:19:39.338073 27187 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:39.338411 27187 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.339148 27187 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.191679 27093 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:39.263860 27200 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240426 13:19:39.314481 27173 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:39.183596 26983 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.262948 26977 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.709s user 0.010s sys 0.000s
I20240426 13:19:39.309840 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.219041 27038 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.305078 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.260462 26721 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20240426 13:19:39.303963 27197 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:39.318769 26993 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.585s user 0.008s sys 0.000s
I20240426 13:19:39.309057 26839 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.324450 26996 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:39.275184 27193 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.500s user 0.002s sys 0.009s
I20240426 13:19:39.303752 27176 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.304173 27141 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.390918 24953 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:39.337317 27157 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.340646 27093 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.371774 27187 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:39.365991 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 56259.2us
I20240426 13:19:39.307265 27035 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20240426 13:19:39.303248 27107 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.398258 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:39.388556 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 82624.6us
W20240426 13:19:39.301465 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 131662us
W20240426 13:19:39.392249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48057.4us
I20240426 13:19:39.368705 27084 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.372200 27173 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.395864 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:39.390007 27165 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.401099 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.410086 24952 raft_consensus.cc:2362] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5. Re-sending same reply.
I20240426 13:19:39.406960 25104 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.372555 26964 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240426 13:19:39.416043 27165 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:39.416401 27165 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.417076 27165 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.417992 26945 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:39.418247 27165 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:39.418977 26945 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:19:39.420222 26945 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:39.420861 26945 consensus_queue.cc:1225] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:39.421047 25588 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.421320 26945 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:39.421831 26945 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.393518 27114 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:39.422328 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:39.422577 26945 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.423434 27209 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.424329 26857 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.424819 26857 raft_consensus.cc:686] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:39.425599 26857 consensus_queue.cc:237] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.387667 26993 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.428314 26993 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:39.431730 25028 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:39.433497 26824 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:39.434644 26824 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:39.435001 26824 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.436121 26824 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.449800 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.437211 26824 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:39.457422 24947 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:39.394950 27157 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.422601 27114 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.451666 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.460178 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.461195 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.462543 25100 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.449380 26839 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.464499 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:39.465266 26413 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:39.465623 26413 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:39.452340 27036 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.465914 26413 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.455549 27084 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:39.467216 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.468066 27016 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240426 13:19:39.453290 27035 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20240426 13:19:39.459558 26945 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:39.472380 27205 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240426 13:19:39.474711 26773 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:39.480257 26713 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:39.481211 24950 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:39.488952 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.492254 26413 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.461072 26983 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:39.498535 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.521569 25031 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:39.506459 25104 raft_consensus.cc:1261] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:39.522114 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.522621 26413 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.523576 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.524189 25104 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.530671 25028 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:39.536942 25104 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.538152 24996 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:39.541132 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:39.542076 25620 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:39.542918 27061 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.464534 27146 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20240426 13:19:39.545341 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:39.546705 27016 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:39.547101 27016 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.547557 27016 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.548884 26861 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:39.497830 26839 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240426 13:19:39.512230 27209 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20240426 13:19:39.462564 27040 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20240426 13:19:39.561439 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:39.564726 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.565672 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:39.520522 26713 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:39.554723 26993 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:39.592268 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.561288 27150 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:39.550662 27084 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240426 13:19:39.619290 27107 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.619410 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:39.512909 27173 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.620316 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.621941 26965 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.622556 26965 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.585680 26983 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:39.624240 27184 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:39.565948 26964 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20240426 13:19:39.622556 27186 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:39.625231 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.625506 27186 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.625648 26993 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.626006 27186 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.626533 26993 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.623736 26965 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.624264 25621 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 1 LEARNER]: Advancing to term 2
I20240426 13:19:39.597493 27036 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:39.600275 27157 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.601509 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.632714 25098 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.588526 27115 consensus_queue.cc:1035] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240426 13:19:39.551585 27114 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.607944 26413 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:39.574468 27038 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:39.586845 27035 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:39.634632 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:39.637620 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.678210 26413 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:39.681263 26413 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:39.683173 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:39.695231 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.695856 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.696626 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.697176 24952 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.700644 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:39.701243 24948 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.703661 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.704396 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:39.704710 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.706105 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.639797 26987 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20240426 13:19:39.631563 27162 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 13:19:39.686251 27035 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.712294 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:39.712426 27035 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.714680 27035 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.700287 27102 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:39.720180 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:39.721097 25101 raft_consensus.cc:2396] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:39.722806 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.652199 27187 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:39.723778 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:39.649276 27038 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.727512 25624 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:39.725010 24955 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.733011 24952 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.687326 27107 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:39.761168 25101 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:39.758155 24948 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.764961 26413 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:39.757582 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:39.767081 25624 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:39.687623 26839 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.645278 27157 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.078s
I20240426 13:19:39.726512 27187 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:39.832158 25624 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:39.758646 26969 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.757915 26757 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:39.760123 27107 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:39.801762 27187 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:39.839730 26679 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:39.814800 26839 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:39.913674 25624 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:39.890244 26977 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:39.892580 26757 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.872093 26969 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.841393 27186 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:39.940659 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 177882us
W20240426 13:19:39.948661 27223 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
W20240426 13:19:39.948763 27226 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
W20240426 13:19:39.950556 27221 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240426 13:19:39.950932 27102 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:39.952629 26969 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:39.954350 25029 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:39.955539 25030 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:39.958725 25029 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:39.961596 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:39.910825 26679 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.926323 26824 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20240426 13:19:39.967330 26839 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:39.897418 27038 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:39.987191 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:39.929298 26977 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:39.993804 25029 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:40.001050 24956 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:40.001431 24956 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 13:19:40.011101 24956 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:39.926028 27107 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.030079 24948 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20240426 13:19:40.025457 27236 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20240426 13:19:40.000077 27225 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.105s user 0.001s sys 0.001s
W20240426 13:19:40.009028 27224 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.288s user 0.001s sys 0.000s
W20240426 13:19:40.036829 26945 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.516s user 0.002s sys 0.006s
W20240426 13:19:40.037693 27176 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.645s user 0.010s sys 0.000s
I20240426 13:19:40.037998 27186 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.038101 27176 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:40.039598 25102 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
W20240426 13:19:40.024819 27241 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240426 13:19:39.977627 27163 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20240426 13:19:40.018312 27244 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:40.020519 27228 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20240426 13:19:39.988530 27035 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240426 13:19:40.052822 25103 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:40.055354 25100 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:40.024602 27016 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.060220 27016 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.029547 26773 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240426 13:19:40.066218 25024 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:40.066810 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:40.068398 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.070688 25028 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:40.071836 25022 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.073042 24948 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:40.071328 25023 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.076783 24948 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:40.077270 27147 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:40.082778 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:40.080777 27147 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:40.086572 26977 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.057488 26757 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.038723 27246 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240426 13:19:40.099450 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:40.100983 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:40.108777 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.109426 25624 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:40.110419 25103 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:40.110755 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.113622 26993 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:40.113754 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:40.114208 26993 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:40.115048 27242 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 13:19:40.115226 26993 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.051291 27244 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.112085 27186 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:40.106194 27147 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.314445 27190 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20240426 13:19:40.332366 27244 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:40.300244 27234 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20240426 13:19:40.109087 27038 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.116888 26679 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:40.312952 27175 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
W20240426 13:19:40.302242 27257 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20240426 13:19:40.314697 27187 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.362376 27038 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:40.368424 27061 consensus_queue.cc:1159] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 4 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:40.368261 27147 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.383390 27051 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:40.387676 27051 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:40.388594 27051 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.389791 27038 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.385749 27187 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.469231 27268 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.492291 27268 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.472649 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.474835 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.494279 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:40.494488 24950 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:40.474853 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:40.487587 27271 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.471102 27269 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.495887 27268 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.502979 25027 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.503180 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.518816 25024 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:40.504619 25026 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:40.500260 27269 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.499876 27271 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.515730 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:40.545704 26679 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:40.566243 27035 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.567597 27271 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.567379 27035 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.569758 27268 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.570447 27106 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.570519 27035 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.574273 27171 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.575059 27016 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.546852 26679 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.570776 27268 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.146124 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.623958 27268 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.625458 26964 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.626592 26964 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.628232 26839 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.629298 26839 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.637156 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.638338 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:40.638903 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.639101 27102 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.644774 25104 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:40.646437 27238 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.728s user 0.012s sys 0.006s
I20240426 13:19:40.648520 27269 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.652590 27248 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:40.653108 27248 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.653553 27248 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.654847 27102 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.660221 26713 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.662062 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 146795us
I20240426 13:19:40.662438 27253 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:40.663002 27253 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.661502 27102 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.665097 27253 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:40.719841 27260 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.720791 27260 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:40.723925 27232 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20240426 13:19:40.735841 27016 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:40.723961 27260 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.744302 25022 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:40.748090 26824 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.748800 26824 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.749424 26679 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.749907 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:40.750901 26824 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.716770 26839 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.711954 27171 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.755070 24951 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:40.738852 27016 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.758872 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:40.715386 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 197504us
I20240426 13:19:40.768436 25100 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:40.768733 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.769675 25022 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:40.780623 25105 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.666s user 0.000s sys 0.013s
I20240426 13:19:40.781258 25105 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:19:40.793641 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.005s sys 0.029s
W20240426 13:19:40.789479 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.019s sys 0.017s
W20240426 13:19:40.797886 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.006s sys 0.029s
I20240426 13:19:40.798151 25031 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
W20240426 13:19:40.801822 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.019s sys 0.018s
W20240426 13:19:40.800347 24996 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.004s sys 0.048s
W20240426 13:19:40.804565 24996 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.005s sys 0.048s
W20240426 13:19:40.806461 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 705533us
W20240426 13:19:40.806614 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50771.6us
I20240426 13:19:40.809211 27282 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.809929 27282 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.811556 24950 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:40.814011 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:40.818928 27038 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.821228 27085 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:40.821331 26679 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:40.825524 27020 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:40.868362 25104 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:40.826270 27093 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:40.830178 27171 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.871078 26824 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.871527 26824 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.871601 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:40.872962 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58334.3us
I20240426 13:19:40.874419 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:40.874568 27282 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.839150 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.876698 25624 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:40.866431 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52498.8us
I20240426 13:19:40.877362 27038 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:40.877977 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:40.878273 25100 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:19:40.840605 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.879153 27284 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:40.879796 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.879987 27284 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:40.881422 27016 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:40.871897 26824 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.881953 27016 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:40.883109 27016 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.883337 27248 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:40.884186 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80390.8us
I20240426 13:19:40.884521 27165 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.885200 27165 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:40.885677 27248 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.886524 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:40.887202 27165 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.887975 25103 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:40.889230 25024 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:40.889461 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.890018 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.890123 24947 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:40.869315 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 752518us
I20240426 13:19:40.846232 27085 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:40.897677 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.900600 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.894246 27248 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.902001 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:40.896735 26679 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:40.903082 26679 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:40.903493 26679 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:40.905884 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.897838 27253 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.912329 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.913215 25022 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.914707 27085 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.915958 27085 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:40.920563 25100 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:40.920923 26679 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.923362 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:40.923485 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.925388 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.926236 25028 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:40.926856 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.932590 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.895247 27165 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.926221 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.936079 24952 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:40.925711 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:40.942070 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 134197us
I20240426 13:19:40.944478 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:40.946888 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.949995 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:40.951216 27150 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:40.951645 27150 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.952047 27150 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:40.948747 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.943672 25101 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:40.957954 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.959215 27115 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:40.959700 27115 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:40.960225 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.447824 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:40.960106 27115 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.962162 27165 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.966449 27141 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:40.967118 27141 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.968207 25031 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:40.969125 27141 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.971994 27106 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:40.972223 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1368) took 1252 ms. Trace:
I20240426 13:19:40.972741 25027 rpcz_store.cc:276] 0426 13:19:39.719687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.722575 (+ 2888us) service_pool.cc:225] Handling call
0426 13:19:40.971990 (+1249415us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:40.972136 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1791739136}
I20240426 13:19:40.962510 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:40.975317 27276 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:40.977037 27146 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.977710 27146 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.980978 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:40.981334 27163 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:40.981680 24954 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 1.
I20240426 13:19:40.981951 27163 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.982893 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:40.983060 25100 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:40.984184 27253 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.986845 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:40.972779 27106 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:40.930358 27248 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:40.997704 25069 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:40.997095 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:41.009222 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.009577 27186 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:41.012681 27108 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.012799 27162 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:41.013394 27162 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.014047 27162 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:41.013461 27108 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.016876 25029 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:41.026093 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84308us
I20240426 13:19:40.994782 27146 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:40.994623 27163 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:40.971999 27165 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.054366 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.047976 27276 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:41.060134 24954 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:41.062885 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:41.063067 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.059616 27141 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.064118 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.064474 27141 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:41.066251 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1493) took 2095 ms (client timeout 2206 ms). Trace:
I20240426 13:19:41.064888 25023 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:41.067790 27141 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.240706 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.241434 27106 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.078634 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:41.191884 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 133488us
I20240426 13:19:41.228797 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1519) took 1083 ms. Trace:
I20240426 13:19:41.243557 27107 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:41.243700 25103 rpcz_store.cc:276] 0426 13:19:40.145482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.145806 (+ 324us) service_pool.cc:225] Handling call
0426 13:19:41.228534 (+1082728us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400384fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:41.228714 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":511661824}
I20240426 13:19:41.061534 26679 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.245862 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 186121us
I20240426 13:19:41.245666 27150 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:41.235751 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 284) took 1178 ms. Trace:
I20240426 13:19:41.227355 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.269857 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:41.271800 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 202154us
I20240426 13:19:41.074929 27084 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:41.283855 27296 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.187s user 0.003s sys 0.000s
I20240426 13:19:41.283689 27084 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.266973 25026 rpcz_store.cc:276] 0426 13:19:40.057397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.057559 (+ 162us) service_pool.cc:225] Handling call
0426 13:19:40.074637 (+ 17078us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:40.520744 (+446107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.235405 (+714661us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.235654 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986973056}
I20240426 13:19:41.093461 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.168720 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:41.307718 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.077598 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:41.166956 27283 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20240426 13:19:41.077595 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.314752 24956 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.232686 27272 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.321306 24947 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:41.321975 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1469) took 1205 ms. Trace:
I20240426 13:19:41.322533 24948 rpcz_store.cc:276] 0426 13:19:40.079998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.080184 (+ 186us) service_pool.cc:225] Handling call
0426 13:19:40.967761 (+887577us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:40.967976 (+ 215us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.968322 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.968346 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.968787 (+ 441us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.011010 (+ 42223us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:41.011050 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.011113 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.282960 (+271847us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:41.285113 (+ 2153us) raft_consensus.cc:1671] finished
0426 13:19:41.285138 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.285275 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6558,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":190607488,"thread_start_us":37995,"threads_started":1,"wal-append.queue_time_us":127724}
I20240426 13:19:41.074748 26721 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.324955 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:41.325567 26721 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.322046 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:41.328750 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1508) took 2035 ms. Trace:
I20240426 13:19:41.329231 26979 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.329118 25102 rpcz_store.cc:276] 0426 13:19:39.292994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.299361 (+ 6367us) service_pool.cc:225] Handling call
0426 13:19:39.306115 (+ 6754us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:40.039962 (+733847us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.313049 (+1273087us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.328579 (+ 15530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1686546304}
I20240426 13:19:41.329818 26979 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.331457 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:41.327942 26721 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.332546 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1736) took 1723 ms. Trace:
I20240426 13:19:41.333117 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1821) took 1426 ms. Trace:
I20240426 13:19:41.332230 27176 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.096372 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:41.333843 27176 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.334511 25615 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.339288 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 497) took 1842 ms. Trace:
I20240426 13:19:41.233207 27248 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.321375 27281 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.230s user 0.011s sys 0.050s
I20240426 13:19:41.331748 26979 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.316488 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.332047 25022 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:41.096457 27267 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:41.066687 25099 rpcz_store.cc:269] 0426 13:19:38.970283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:38.970430 (+ 147us) service_pool.cc:225] Handling call
0426 13:19:41.065904 (+2095474us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:41.066041 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2235574144}
I20240426 13:19:41.092267 27093 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.181257 26964 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.441031 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:41.442255 26964 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.443259 27084 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:41.442965 26964 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.445621 27248 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.446170 27248 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.445680 27210 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.448300 26965 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.448360 27210 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.449120 26965 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.450214 26679 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:41.451172 26965 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.452601 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.453172 27210 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.073036 26945 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.454329 27248 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:41.454254 26945 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.455873 27291 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:41.455984 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 85641.5us
I20240426 13:19:41.456396 26945 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.456539 27291 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.321867 27272 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.458701 27291 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:41.441097 26824 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.559s user 0.008s sys 0.002s
I20240426 13:19:41.459203 27291 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.333559 25030 rpcz_store.cc:276] 0426 13:19:39.906360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.906498 (+ 138us) service_pool.cc:225] Handling call
0426 13:19:39.924880 (+ 18382us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:39.972431 (+ 47551us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.332713 (+1360282us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:41.332967 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69971200}
I20240426 13:19:41.068090 27061 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.424s
I20240426 13:19:41.450837 26679 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.167688 27271 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.402693 25031 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:41.335937 27176 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:41.465669 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 157028us
I20240426 13:19:41.333254 27288 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.195s
I20240426 13:19:41.459546 26824 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.332921 24953 rpcz_store.cc:276] 0426 13:19:39.562492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.562662 (+ 170us) service_pool.cc:225] Handling call
0426 13:19:39.562982 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:39.572454 (+ 9472us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:40.652952 (+1080498us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b180029bf60. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda93cedad7 00007fda90321997 00007fda9033109f 00007fda90331018
0426 13:19:40.800017 (+147065us) log.cc:842] Serialized 14 byte log entry
0426 13:19:40.961996 (+161979us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:41.007658 (+ 45662us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.007814 (+ 156us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:41.008384 (+ 570us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.048190 (+ 39806us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.056743 (+ 8553us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.273312 (+216569us) raft_consensus.cc:1671] finished
0426 13:19:41.273344 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.285657 (+ 12313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62250,"prepare.queue_time_us":345511,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":65690,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":22737,"spinlock_wait_cycles":5281152,"thread_start_us":200482,"threads_started":2,"wal-append.queue_time_us":262441}
I20240426 13:19:41.490797 25615 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:41.474143 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.307103 27162 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.475545 24956 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:41.340085 27282 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.225s
I20240426 13:19:41.340917 27106 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:41.471529 27298 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.373s user 0.002s sys 0.000s
I20240426 13:19:41.544664 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:42.365685 27111 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.938326 27276 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:42.258525 27016 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.566051 27171 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:41.458701 27272 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:41.772064 25028 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.052s sys 0.085s
W20240426 13:19:41.470700 27289 log.cc:924] Time spent T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.413s user 0.002s sys 0.000s
W20240426 13:19:41.565595 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 404191us
I20240426 13:19:42.371292 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 324) took 1610 ms. Trace:
I20240426 13:19:42.371641 24955 rpcz_store.cc:276] 0426 13:19:40.760669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.760817 (+ 148us) service_pool.cc:225] Handling call
0426 13:19:40.761261 (+ 444us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.761774 (+ 513us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.761808 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.762275 (+ 467us) log.cc:842] Serialized 26 byte log entry
0426 13:19:40.791142 (+ 28867us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:40.791192 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:40.791270 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.469991 (+678721us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:42.370920 (+900929us) raft_consensus.cc:1671] finished
0426 13:19:42.370941 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.371133 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1588,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":7772,"spinlock_wait_cycles":4600576,"thread_start_us":28384,"threads_started":1,"wal-append.queue_time_us":170727}
I20240426 13:19:41.564731 26977 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.373473 26977 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.367837 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:42.150417 27280 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:41.772274 27271 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.374946 26912 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:41.549755 27114 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:42.377564 27280 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.568362 26965 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.469008 27093 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.568578 27186 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.412506 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:41.562366 27108 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:41.550446 27267 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.572803 27051 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:41.549577 27203 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.567904 27273 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:42.149596 27107 consensus_meta.cc:287] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.893s user 0.007s sys 0.004s
I20240426 13:19:41.565090 27284 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:41.999868 27247 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.369027 27016 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.367503 27111 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:41.459823 27291 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:41.905151 26979 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:41.833771 27252 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:42.465373 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 900176us
I20240426 13:19:42.387454 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1748) took 1495 ms. Trace:
I20240426 13:19:41.832824 27136 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.388657 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:41.753063 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.162942 27176 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:42.369918 25028 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s user 0.078s sys 0.114s
I20240426 13:19:42.393291 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:42.368404 27085 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:42.367357 27276 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:41.548501 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:41.777643 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.403708 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1753) took 1318 ms. Trace:
W20240426 13:19:41.735208 27115 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.758s user 0.001s sys 0.008s
I20240426 13:19:42.405014 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:41.473104 27150 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.366221 26713 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:42.411464 25102 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 1. Current term is 2. Ops: []
W20240426 13:19:42.412395 27113 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.069s user 0.014s sys 0.014s
I20240426 13:19:42.403168 26679 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.404851 27171 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.420930 24951 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:42.428282 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.369307 27106 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:42.420269 27038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.970s user 0.041s sys 0.078s
I20240426 13:19:42.418402 27108 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:42.496239 25022 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.784s user 0.005s sys 0.006s
I20240426 13:19:42.523455 25022 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.524015 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1842) took 1203 ms. Trace:
I20240426 13:19:42.524446 25022 rpcz_store.cc:276] 0426 13:19:41.320560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.320728 (+ 168us) service_pool.cc:225] Handling call
0426 13:19:42.523958 (+1203230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:42.523087 27038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s user 0.042s sys 0.078s
I20240426 13:19:42.498042 24947 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:42.525187 27108 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.525382 27136 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.526125 26839 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.526702 26839 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.527257 27108 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.527464 27136 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.419649 27272 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.501300 24956 rpcz_store.cc:276] 0426 13:19:41.075690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.077226 (+ 1536us) service_pool.cc:225] Handling call
0426 13:19:42.393973 (+1316747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:42.412951 27186 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:42.507148 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 959108us
W20240426 13:19:42.514879 27113 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.180s user 0.014s sys 0.014s
I20240426 13:19:42.509143 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1383) took 1563 ms. Trace:
I20240426 13:19:42.466076 24949 rpcz_store.cc:276] 0426 13:19:40.892010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.892177 (+ 167us) service_pool.cc:225] Handling call
0426 13:19:40.896177 (+ 4000us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.931587 (+ 35410us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.931615 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.954549 (+ 22934us) log.cc:842] Serialized 26 byte log entry
0426 13:19:41.057285 (+102736us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:41.330360 (+273075us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.470871 (+140511us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:41.564841 (+ 93970us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.564922 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:42.387179 (+822257us) raft_consensus.cc:1671] finished
0426 13:19:42.387279 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.387400 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":452436,"prepare.queue_time_us":273102,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":235489,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":6362,"spinlock_wait_cycles":151641984,"thread_start_us":80591,"threads_started":1,"wal-append.queue_time_us":121204}
I20240426 13:19:42.501332 25099 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:42.518275 25097 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:42.502900 27107 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.527834 27108 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:42.531689 24947 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:42.547432 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 46035.9us
I20240426 13:19:42.461361 27284 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.530366 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.529515 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:42.539995 26839 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.544974 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.549655 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.495260 27247 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.549916 27171 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.434736 26912 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.500921 27273 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.511132 27252 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.514530 26979 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.535501 25028 rpcz_store.cc:276] 0426 13:19:40.945826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.945928 (+ 102us) service_pool.cc:225] Handling call
0426 13:19:41.246708 (+300780us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:41.246951 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:41.247313 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:41.247663 (+ 350us) log.cc:842] Serialized 14 byte log entry
0426 13:19:42.508716 (+1261053us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:42.508786 (+ 70us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:42.508853 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.508937 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:42.509079 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1288,"prepare.queue_time_us":750,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"spinlock_wait_cycles":3084544,"thread_start_us":1249864,"threads_started":1}
I20240426 13:19:42.566063 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.528493 27150 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.515565 27115 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.549497 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:42.537304 27259 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.532013 26679 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.579337 27186 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.580216 26964 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.529829 27176 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.583518 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1523) took 1127 ms. Trace:
I20240426 13:19:42.608340 25102 rpcz_store.cc:276] 0426 13:19:41.455869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.460499 (+ 4630us) service_pool.cc:225] Handling call
0426 13:19:41.469322 (+ 8823us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.518197 (+1048875us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.583289 (+ 65092us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:42.583464 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2164877696}
I20240426 13:19:42.528038 27136 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.587172 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.587442 27150 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.589154 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.513074 27203 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.549721 26969 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.592752 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.594521 27093 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.594579 27115 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.594892 27174 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:42.595562 25104 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:42.613250 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:42.596422 27291 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.596597 26979 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.596648 27171 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.550071 26656 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.607623 25622 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:42.609853 24952 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:42.548408 27107 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.589655 27108 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.597170 27259 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.618254 27150 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.617950 27136 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.594662 26713 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.566948 27272 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.615489 26964 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.500155 26965 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.608520 27162 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.638377 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.615865 26945 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.616364 26679 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.643823 26979 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.647825 24949 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:42.659145 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.636076 27186 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.643328 27171 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:42.639765 27271 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.646920 27174 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:42.681332 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:42.672859 24954 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:42.651170 27197 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.637992 26996 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20240426 13:19:42.615926 26969 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.692852 24954 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:41.339823 25621 rpcz_store.cc:276] 0426 13:19:39.496518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:39.501887 (+ 5369us) service_pool.cc:225] Handling call
0426 13:19:39.591094 (+ 89207us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:40.546011 (+954917us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:40.546079 (+ 68us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:40.546940 (+ 861us) log.cc:842] Serialized 26 byte log entry
0426 13:19:40.786955 (+240015us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:40.902593 (+115638us) log.cc:842] Serialized 14 byte log entry
0426 13:19:41.020045 (+117452us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:41.051032 (+ 30987us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:41.338869 (+287837us) raft_consensus.cc:1671] finished
0426 13:19:41.338904 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:41.339081 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66431,"prepare.queue_time_us":150233,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":233086,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":253,"spinlock_wait_cycles":9202560,"thread_start_us":666715,"threads_started":3,"wal-append.queue_time_us":389498,"wal-append.run_cpu_time_us":7733,"wal-append.run_wall_time_us":1395081}
I20240426 13:19:42.668982 26839 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.672299 27162 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.686749 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:42.697295 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:42.690054 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81912us
I20240426 13:19:42.693987 25023 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:42.684295 24947 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:42.697945 25615 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:42.668483 27113 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.580322 27280 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.593401 27247 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.711854 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:42.691217 27174 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:42.610374 27267 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.670181 26713 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.692215 27016 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.680968 26656 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.720114 27273 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.721159 27136 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.695935 27186 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.716248 27106 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.722615 27176 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.718266 27174 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:42.723378 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:42.725039 27085 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.726581 25100 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:42.685076 26979 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.728312 25616 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:42.729409 26969 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.732105 27162 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.695537 27271 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.739423 27302 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.733726 27272 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.718626 27113 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.691130 27111 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.673038 27108 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.739449 25105 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:42.682402 26945 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.716352 27280 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:42.765762 27106 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:42.852164 27106 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.696475 26964 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.732908 27259 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.763247 26987 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:42.704110 27171 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:42.820734 25022 raft_consensus.cc:1261] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:42.824955 24949 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:42.709578 26839 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.767756 27252 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:42.762259 26912 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.774960 27203 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:42.716719 27197 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.937109 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:43.055207 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 118236us
I20240426 13:19:42.987574 26679 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.056104 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.056779 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:42.952564 27107 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:42.992635 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1360) took 2070 ms (client timeout 2129 ms). Trace:
I20240426 13:19:42.781533 27085 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.786365 26996 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.060882 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 360) took 1742 ms. Trace:
I20240426 13:19:43.061568 26839 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.057297 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:43.061803 25624 rpcz_store.cc:276] 0426 13:19:41.318056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.320839 (+ 2783us) service_pool.cc:225] Handling call
0426 13:19:41.415715 (+ 94876us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:42.461141 (+1045426us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.461183 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:42.520736 (+ 59553us) log.cc:842] Serialized 52 byte log entry
0426 13:19:42.958410 (+437674us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:42.958465 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:42.958564 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.060653 (+102089us) raft_consensus.cc:1671] finished
0426 13:19:43.060672 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.060819 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":925138,"raft.queue_time_us":998797,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":36130,"thread_start_us":391994,"threads_started":1,"wal-append.queue_time_us":524477}
I20240426 13:19:43.063162 27085 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.058684 27106 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.063235 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.064030 26996 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.064616 25100 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:42.858731 27302 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.066291 26996 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:42.962352 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 316634us
I20240426 13:19:43.066901 27302 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.056861 24947 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:43.056798 26679 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:43.069383 27197 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.069810 26679 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:43.070472 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.047s sys 0.054s
W20240426 13:19:43.070912 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.047s sys 0.054s
I20240426 13:19:43.070860 26679 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.068621 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:42.901000 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.000512 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:43.074040 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 173134us
W20240426 13:19:43.074802 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 74399.4us
I20240426 13:19:43.074946 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:43.061264 25104 rpcz_store.cc:269] 0426 13:19:40.921837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.932314 (+ 10477us) service_pool.cc:225] Handling call
0426 13:19:42.992404 (+2060090us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:42.992560 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3810614528}
I20240426 13:19:43.075837 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:42.828593 27253 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.089686 25104 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:43.071759 27106 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.088866 27305 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.058223 26979 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:42.858469 26987 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:42.951895 27176 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.088913 27162 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.072091 26912 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.120200 27113 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.129215 26912 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.129781 26977 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.089612 27272 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:43.120177 27147 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:43.120910 27114 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:43.143929 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:43.121068 27300 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.424s user 0.000s sys 0.000s
I20240426 13:19:43.121181 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.080379 26839 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:43.128764 27162 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:43.073313 27186 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.130378 26656 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.140388 27252 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.141167 27093 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:43.141917 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 73366us
I20240426 13:19:42.937624 27271 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:42.901597 27280 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.131153 27106 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.242008 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.242846 25620 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:43.243362 27271 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:43.245306 27301 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.550s user 0.000s sys 0.001s
I20240426 13:19:43.245589 27280 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.246160 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.185564 27114 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.247819 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1482) took 1884 ms. Trace:
W20240426 13:19:43.225994 25097 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.685s user 0.005s sys 0.003s
I20240426 13:19:43.180274 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.250985 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 176147us
I20240426 13:19:43.180429 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.252485 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.248654 25097 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:43.221151 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:43.120935 27061 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:43.187500 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.254734 25097 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1525) took 1505 ms (client timeout 1898 ms). Trace:
I20240426 13:19:43.187475 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.256851 25615 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
W20240426 13:19:43.257535 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 113721us
I20240426 13:19:43.254294 27249 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.259370 27252 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.259477 26987 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.260354 27249 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.260844 26987 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.261651 27252 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.262368 27249 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:43.255636 25097 rpcz_store.cc:269] 0426 13:19:41.749367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.749643 (+ 276us) service_pool.cc:225] Handling call
0426 13:19:43.254668 (+1505025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:43.106979 27253 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.255919 25027 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:43.113312 27107 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.263666 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:43.248184 24950 rpcz_store.cc:276] 0426 13:19:41.363364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.634050 (+1270686us) service_pool.cc:225] Handling call
0426 13:19:42.648585 (+ 14535us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:42.661676 (+ 13091us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.661702 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:42.684885 (+ 23183us) log.cc:842] Serialized 26 byte log entry
0426 13:19:43.015838 (+330953us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:43.015890 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.015969 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.247601 (+231632us) raft_consensus.cc:1671] finished
0426 13:19:43.247634 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.247758 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12731,"raft.queue_time_us":32283,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":11312,"thread_start_us":318653,"threads_started":1,"wal-append.queue_time_us":424642}
I20240426 13:19:43.247211 27162 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.271360 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:43.259317 26964 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.173058 26979 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.272717 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.172878 27038 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:43.224262 27171 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.164111 26969 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:43.142855 27272 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:43.187507 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.220052 26713 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.254127 26839 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:43.188937 27303 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.263219 26977 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:43.283931 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.292061 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:43.263830 27106 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.295622 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.293665 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.297379 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.265470 27016 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.299160 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.268201 27108 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.300211 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:43.299870 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 224119us
I20240426 13:19:43.300850 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.269922 27162 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.282521 27271 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:43.304747 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:43.376348 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1760) took 1474 ms. Trace:
I20240426 13:19:43.270323 27249 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
W20240426 13:19:43.377287 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 104100us
I20240426 13:19:43.164072 27176 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.164207 27147 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.379400 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.291082 27311 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:43.262869 27093 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.292786 26979 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.301281 27272 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:43.294571 27107 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.379112 26969 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.382311 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.383026 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.384761 24952 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:43.380896 25031 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:43.385434 26977 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:43.385969 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94000us
I20240426 13:19:43.386090 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.395457 24949 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:43.417161 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:43.420424 24950 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.420325 24953 rpcz_store.cc:276] 0426 13:19:41.902285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.908524 (+ 6239us) service_pool.cc:225] Handling call
0426 13:19:43.304278 (+1395754us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:43.304454 (+ 176us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.304647 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.375996 (+ 71349us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.376058 (+ 62us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:43.376097 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.376177 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.376290 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18335104}
I20240426 13:19:43.122990 27305 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.375891 26839 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.444839 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.451958 26912 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.473177 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:43.489907 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.536320 25027 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.536513 24947 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:19:43.453132 27276 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.929s user 0.010s sys 0.000s
W20240426 13:19:43.546634 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 118141us
I20240426 13:19:43.422837 27271 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:43.550505 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 133476us
I20240426 13:19:43.556160 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.572281 27271 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:43.572944 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.573786 25031 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:43.574894 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:43.577244 27174 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.814s user 0.005s sys 0.005s
I20240426 13:19:43.578028 27174 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.532869 26977 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.580789 26977 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.583087 27107 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.504789 27114 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.584853 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.557518 27305 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.558192 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.489825 27311 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.560636 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.565024 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.552258 27162 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.588241 27176 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.506045 27147 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:43.591362 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 327421us
I20240426 13:19:43.595858 24950 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:43.602527 25025 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:43.602535 27303 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.610208 27303 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.611500 24996 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.572121 27203 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:43.610941 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.596877 27197 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:43.458382 27016 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.611111 27174 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.636968 27176 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240426 13:19:43.641469 24920 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:43.589280 27276 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.647855 27311 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:43.651109 27016 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.656311 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:43.657294 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.657927 27323 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:43.665287 27284 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.666805 26979 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.668395 27107 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:43.779592 27176 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:43.669209 26969 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.616701 27203 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.780416 27203 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:43.739826 25069 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.808s (SENT) [suppressed 14 similar messages]
W20240426 13:19:43.777554 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 120344us
I20240426 13:19:43.829264 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.833002 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:43.825892 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 548) took 2881 ms (client timeout 3135 ms). Trace:
W20240426 13:19:43.830034 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 229205us
I20240426 13:19:43.837324 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:43.836973 25622 rpcz_store.cc:269] 0426 13:19:40.946922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.949712 (+ 2790us) service_pool.cc:225] Handling call
0426 13:19:43.825680 (+2875968us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:43.825818 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3789052160}
I20240426 13:19:43.828941 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:43.841907 25104 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:43.842213 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:43.842733 25029 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:43.785431 27162 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:43.640426 27197 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:43.786535 26987 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.525s user 0.009s sys 0.001s
I20240426 13:19:43.819097 26977 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.829170 27108 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:43.779646 27107 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:43.860714 25589 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:43.860821 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.861248 25589 leader_election.cc:401] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240426 13:19:43.834066 27176 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.861557 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:43.847360 26679 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:43.843763 27147 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:43.864478 24955 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1761) took 1937 ms (client timeout 1864 ms). Trace:
W20240426 13:19:43.866854 24955 rpcz_store.cc:269] 0426 13:19:41.926896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.373271 (+446375us) service_pool.cc:225] Handling call
0426 13:19:43.860672 (+1487401us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:43.864399 (+ 3727us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13966080}
I20240426 13:19:43.840498 27093 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.867985 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.815289 27272 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.820947 26839 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.877429 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:43.863698 27107 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:43.863394 27108 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:43.867748 27147 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:43.884271 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:43.873346 27093 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.890627 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:43.899096 26979 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:43.898954 27327 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:43.899662 26979 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:43.900533 26979 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.913385 27272 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:43.920195 27176 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:43.922617 27197 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.925787 27271 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.895606 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1370) took 1122 ms. Trace:
W20240426 13:19:43.935616 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 195835us
I20240426 13:19:43.936770 25099 rpcz_store.cc:276] 0426 13:19:42.772832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.871019 (+1098187us) service_pool.cc:225] Handling call
0426 13:19:43.885220 (+ 14201us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.895163 (+ 9943us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.895206 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.895268 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.895316 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.895397 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.895538 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:43.935698 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 372) took 1198 ms. Trace:
I20240426 13:19:43.943604 25097 rpcz_store.cc:276] 0426 13:19:42.737777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.268382 (+530605us) service_pool.cc:225] Handling call
0426 13:19:43.268670 (+ 288us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.290983 (+ 22313us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.291002 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:43.380686 (+ 89684us) log.cc:842] Serialized 26 byte log entry
0426 13:19:43.745802 (+365116us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:43.746202 (+ 400us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.746424 (+ 222us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:43.935458 (+189034us) raft_consensus.cc:1671] finished
0426 13:19:43.935512 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:43.935642 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1366,"raft.queue_time_us":3226,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6593,"thread_start_us":291981,"threads_started":1,"wal-append.queue_time_us":416869}
I20240426 13:19:43.889652 27147 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:43.951517 25103 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240426 13:19:43.891583 27323 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:43.874902 26912 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20240426 13:19:43.957032 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:43.960587 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1527) took 2163 ms. Trace:
I20240426 13:19:43.960966 26977 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:43.946343 26987 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:43.966265 24996 connection.cc:232] RPC call timeout handler was delayed by -1.2130351069999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
I20240426 13:19:43.966105 25103 rpcz_store.cc:276] 0426 13:19:41.788474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.809726 (+ 21252us) service_pool.cc:225] Handling call
0426 13:19:43.947023 (+2137297us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:43.947282 (+ 259us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.951967 (+ 4685us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:43.952221 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16720768}
W20240426 13:19:43.967644 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 137923us
I20240426 13:19:43.967999 25622 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:43.976663 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.979771 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:43.889866 27305 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20240426 13:19:43.982122 25096 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:43.967664 27271 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:43.986476 27299 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.289s user 0.001s sys 0.008s
I20240426 13:19:43.988313 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.988920 25097 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:43.989694 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:43.990809 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 719501us
I20240426 13:19:43.961560 27108 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:43.991935 27273 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:43.962240 27323 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:43.998340 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1526) took 2203 ms (client timeout 2585 ms). Trace:
I20240426 13:19:43.990831 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:43.999994 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.003428 25103 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:44.001892 25096 rpcz_store.cc:269] 0426 13:19:41.780456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.780582 (+ 126us) service_pool.cc:225] Handling call
0426 13:19:43.976491 (+2195909us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:43.983932 (+ 7441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45856640}
I20240426 13:19:43.967119 27276 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.003049 25097 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:44.011500 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 79029.9us
I20240426 13:19:44.003674 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.019289 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.019956 24950 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:44.020934 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 469220us
I20240426 13:19:44.021056 25029 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:44.021948 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:44.025400 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.025478 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:43.998034 27273 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:44.026551 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.025765 27310 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.000684 27093 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:44.013440 27323 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:44.025439 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.025640 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.028285 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.026362 24953 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.055855 24950 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.033404 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.035418 27171 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.058192 27171 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:44.058358 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.058640 27171 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:44.036101 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.038058 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.039718 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.040148 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.040745 27305 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.041097 27271 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:44.043431 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.027837 26656 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.044606 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.031635 27273 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:44.045689 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1491) took 1341 ms. Trace:
W20240426 13:19:44.045702 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1751) took 2970 ms (client timeout 1808 ms). Trace:
I20240426 13:19:44.049804 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1369) took 1287 ms. Trace:
I20240426 13:19:44.051276 26824 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.057219 24949 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:44.070878 25027 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.516s user 0.007s sys 0.002s
I20240426 13:19:44.220849 25028 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20240426 13:19:44.183204 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146588us
I20240426 13:19:44.227485 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.249737 25031 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:44.222651 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1368) took 1469 ms. Trace:
W20240426 13:19:44.249262 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 209622us
I20240426 13:19:44.155021 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1386) took 1012 ms. Trace:
I20240426 13:19:44.104657 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.262477 25026 rpcz_store.cc:276] 0426 13:19:43.142243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.164277 (+ 22034us) service_pool.cc:225] Handling call
0426 13:19:43.164653 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.556761 (+392108us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.556791 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.556839 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.556871 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.154786 (+597915us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.154835 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.154965 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":901404416}
I20240426 13:19:44.265938 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:44.248425 24951 rpcz_store.cc:269] 0426 13:19:41.075123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.093151 (+ 18028us) service_pool.cc:225] Handling call
0426 13:19:44.045510 (+2952359us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540004b854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.045601 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2551163648}
I20240426 13:19:44.262380 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.247946 24954 rpcz_store.cc:276] 0426 13:19:42.707286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.707409 (+ 123us) service_pool.cc:225] Handling call
0426 13:19:42.718403 (+ 10994us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.986584 (+268181us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:42.986627 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:42.986682 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:42.986722 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.045492 (+1058770us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.045535 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.045634 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616222464}
I20240426 13:19:44.269234 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1857) took 1041 ms. Trace:
W20240426 13:19:44.219581 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1843) took 2780 ms (client timeout 1948 ms). Trace:
W20240426 13:19:44.267935 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1759) took 2383 ms (client timeout 2960 ms). Trace:
I20240426 13:19:44.270128 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.270171 24948 rpcz_store.cc:269] 0426 13:19:41.884236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.884481 (+ 245us) service_pool.cc:225] Handling call
0426 13:19:44.035951 (+2151470us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:44.267874 (+231923us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2663680}
I20240426 13:19:44.059576 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.269606 25027 rpcz_store.cc:276] 0426 13:19:43.235990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.433501 (+197511us) service_pool.cc:225] Handling call
0426 13:19:44.269175 (+835674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:44.245762 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 378) took 1116 ms. Trace:
I20240426 13:19:44.269253 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:44.271241 25023 rpcz_store.cc:269] 0426 13:19:41.439656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.468744 (+ 29088us) service_pool.cc:225] Handling call
0426 13:19:44.219220 (+2750476us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.219466 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":711493376}
I20240426 13:19:44.273530 25102 rpcz_store.cc:276] 0426 13:19:43.129104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.052981 (+923877us) service_pool.cc:225] Handling call
0426 13:19:44.245092 (+192111us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.245328 (+ 236us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.245363 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.245419 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.245465 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.245526 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.245653 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:44.267794 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.265866 24950 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:44.274739 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.061585 24955 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 2: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:44.108994 27263 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:44.286410 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.286692 27263 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:44.287292 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.223165 26912 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:44.224658 27303 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.271466 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.321283 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.216919 26969 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:44.276940 25104 rpcz_store.cc:276] 0426 13:19:42.762506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.847542 (+1085036us) service_pool.cc:225] Handling call
0426 13:19:43.858764 (+ 11222us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.870466 (+ 11702us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.870513 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.870571 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.870620 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.049587 (+178967us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540017f2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.049642 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.049753 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26227584}
I20240426 13:19:44.279760 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.280128 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.325429 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.325704 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:44.285550 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.285710 27093 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.286105 26973 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240426 13:19:44.248350 26656 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:44.334302 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1361) took 3412 ms (client timeout 3135 ms). Trace:
I20240426 13:19:44.272688 27327 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.287864 24950 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:44.314754 25101 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:44.314841 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:44.316179 24921 connection.cc:232] RPC call timeout handler was delayed by -1.739396377999924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
I20240426 13:19:44.153260 27272 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:44.319108 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 303677us
W20240426 13:19:44.319707 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 299453us
I20240426 13:19:44.265726 27249 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:44.322665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.323614 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 289) took 1679 ms. Trace:
I20240426 13:19:44.321933 25025 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:44.325970 24954 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:44.061204 27203 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.326401 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.329124 25026 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:44.268684 27271 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.155087 27171 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.334239 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.060984 26839 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.328954 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.266247 27305 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.194319 27253 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.064123 27310 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.276476 27147 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.335186 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.263973 25098 rpcz_store.cc:276] 0426 13:19:42.753064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.591117 (+838053us) service_pool.cc:225] Handling call
0426 13:19:43.615353 (+ 24236us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.635707 (+ 20354us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:43.840693 (+204986us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.976559 (+135866us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.106396 (+129837us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351
0426 13:19:44.107020 (+ 624us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:44.222019 (+114999us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b18001d1940. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:19:44.222242 (+ 223us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.222301 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:44.222361 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.222444 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.222585 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16998,"prepare.queue_time_us":219565,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":252571,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":4067,"spinlock_wait_cycles":86125952,"thread_start_us":275872,"threads_started":2,"wal-append.queue_time_us":267292}
I20240426 13:19:44.642073 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.643128 25097 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:19:44.642447 25101 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1530) took 2739 ms (client timeout 1811 ms). Trace:
I20240426 13:19:44.335918 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:44.644157 25101 rpcz_store.cc:269] 0426 13:19:41.902634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.902828 (+ 194us) service_pool.cc:225] Handling call
0426 13:19:44.043291 (+2140463us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b5800114f98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:44.642387 (+599096us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15827200}
I20240426 13:19:44.649139 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.334764 25105 rpcz_store.cc:269] 0426 13:19:40.922112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:40.923205 (+ 1093us) service_pool.cc:225] Handling call
0426 13:19:44.334086 (+3410881us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:44.334211 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4171074816}
I20240426 13:19:44.389207 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.666317 24953 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:44.665113 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.389971 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.436969 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.663430 25022 rpcz_store.cc:276] 0426 13:19:42.644693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.654289 (+ 9596us) service_pool.cc:225] Handling call
0426 13:19:42.672933 (+ 18644us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.054455 (+381522us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.323390 (+1268935us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54000564d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.323560 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":338698112}
I20240426 13:19:44.645356 24947 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.668632 25101 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:44.437466 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.572603 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.669684 25619 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.644810 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.436576 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.674459 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.664669 25104 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.675154 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:44.688746 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.689021 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.689479 25023 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:44.689709 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:44.625885 27344 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.298s user 0.000s sys 0.000s
I20240426 13:19:44.697934 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.387992 26656 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.436156 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.703068 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.703190 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.703321 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.704442 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.704700 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.704025 25101 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:44.728333 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.728961 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.705330 27272 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.731724 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:44.706091 27147 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.779304 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.324769 26969 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:44.706343 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 1.
I20240426 13:19:44.707670 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.662796 27327 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.662595 27271 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:44.704337 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.711114 26679 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:44.315701 26912 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:44.641340 27350 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:44.438690 27340 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
W20240426 13:19:44.708017 25622 consensus_meta.cc:287] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.727s user 0.004s sys 0.003s
I20240426 13:19:44.795338 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.708107 27310 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.795656 25622 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.710513 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.721212 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:44.704123 25102 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:44.798630 25619 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.799528 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.745836 27203 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.803014 25102 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:44.328874 27093 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.753847 25104 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:44.794785 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.021s sys 0.088s
W20240426 13:19:44.804430 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.021s sys 0.089s
W20240426 13:19:44.804823 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 530863us
W20240426 13:19:44.775146 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 672239us
I20240426 13:19:44.704279 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:44.710952 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:44.806063 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 95185us
I20240426 13:19:44.806560 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:44.776722 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.777537 25029 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:44.779871 24951 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.807886 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.808610 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.605222 26965 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.790613 27327 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.784262 24954 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:44.785681 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.819489 24955 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.787294 25026 raft_consensus.cc:2354] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:44.787381 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1487) took 2130 ms. Trace:
I20240426 13:19:44.795701 27147 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.796449 26679 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:44.724118 27346 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20240426 13:19:44.791024 25023 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:44.753017 27107 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.864s user 0.009s sys 0.000s
I20240426 13:19:44.323488 27113 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.791812 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.864027 24955 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:44.709884 27305 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.867169 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 383) took 1038 ms. Trace:
I20240426 13:19:44.813824 27339 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:44.794262 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.708225 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.796245 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.710774 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.797586 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.813828 25588 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:44.783293 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.821588 26824 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:44.817235 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.788018 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.815797 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.828155 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.831229 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.788733 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57088.4us
I20240426 13:19:44.874223 25031 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:19:44.876689 24950 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.877048 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.872220 24953 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:44.722365 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:44.867525 25100 rpcz_store.cc:276] 0426 13:19:43.828926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.445274 (+616348us) service_pool.cc:225] Handling call
0426 13:19:44.445614 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.445820 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:44.446081 (+ 261us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.866855 (+420774us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.866904 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.866960 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.867023 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.867113 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33845,"prepare.queue_time_us":153,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":35537,"thread_start_us":386594,"threads_started":1}
I20240426 13:19:44.809130 27093 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:44.751281 27323 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.689s user 0.006s sys 0.000s
W20240426 13:19:44.908288 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94770.7us
I20240426 13:19:44.910125 27353 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:44.907349 27327 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.910635 27353 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:44.911552 27353 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.912675 27327 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:44.914541 27273 consensus_meta.cc:287] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.646s user 0.005s sys 0.001s
I20240426 13:19:44.907469 27323 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.914964 27273 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.916067 24951 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:44.916863 27323 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:44.919133 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.880847 26969 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:44.919876 25097 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:44.920703 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.864053 27150 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:44.817862 26839 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:44.872182 25022 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:44.855028 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.820387 24956 rpcz_store.cc:276] 0426 13:19:42.650038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.650176 (+ 138us) service_pool.cc:225] Handling call
0426 13:19:42.664136 (+ 13960us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:42.671515 (+ 7379us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:43.000235 (+328720us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.271567 (+271332us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.989189 (+717622us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b1800419aa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:19:43.989391 (+ 202us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.989461 (+ 70us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:43.989525 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.043916 (+ 54391us) spinlock_profiling.cc:243] Waited 8.43 ms on lock 0x7b1800419aa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:44.780321 (+736405us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:44.780366 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.780471 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31076,"prepare.queue_time_us":384786,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":737631,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":214,"spinlock_wait_cycles":45950976,"thread_start_us":565884,"threads_started":2,"wal-append.queue_time_us":415887,"wal-append.run_cpu_time_us":1312,"wal-append.run_wall_time_us":1012264}
I20240426 13:19:44.872400 25098 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:44.880148 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:44.869191 27113 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.882472 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.882864 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 374) took 1828 ms. Trace:
I20240426 13:19:44.883318 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.884157 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:44.929682 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54348.4us
I20240426 13:19:44.930119 25101 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:44.884990 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.885588 27263 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.886039 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.933877 25027 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.887025 24953 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:44.887480 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.935166 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:44.888127 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.889070 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 385) took 1021 ms. Trace:
I20240426 13:19:44.936328 25100 raft_consensus.cc:2354] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:44.890209 27253 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:44.890179 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 377) took 1766 ms. Trace:
I20240426 13:19:44.855878 26679 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.909734 27339 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:44.895335 26945 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.898056 27272 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.904578 25620 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.959841 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:44.827061 27147 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:44.868016 27340 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240426 13:19:44.921509 25615 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:44.909384 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:44.969954 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60645.9us
I20240426 13:19:44.924722 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.873003 24948 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:44.931649 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 274839us
W20240426 13:19:44.931524 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4774963779999553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20240426 13:19:44.939376 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.964982 27147 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:44.802577 27350 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:44.936152 25105 rpcz_store.cc:276] 0426 13:19:43.867637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.665408 (+797771us) service_pool.cc:225] Handling call
0426 13:19:44.706638 (+ 41230us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.706820 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:44.707064 (+ 244us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.888741 (+181677us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:44.888798 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:44.888841 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.888906 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.889012 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36154,"prepare.queue_time_us":3283,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":36919,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":145039,"threads_started":1}
I20240426 13:19:44.943387 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.946545 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:44.949271 27113 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.951097 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 338) took 1685 ms. Trace:
I20240426 13:19:44.951123 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.952445 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.971680 25102 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:44.980692 25098 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:44.980800 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 179961us
I20240426 13:19:44.953061 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.957645 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.942104 26839 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.958945 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.869776 27107 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:44.983784 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.963198 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.962167 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.984968 25026 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:44.985028 25096 raft_consensus.cc:2396] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:44.929350 25099 rpcz_store.cc:276] 0426 13:19:43.054780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.943435 (+888655us) service_pool.cc:225] Handling call
0426 13:19:43.960442 (+ 17007us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:43.960761 (+ 319us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:43.960794 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:43.961241 (+ 447us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.027490 (+ 66249us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:44.027530 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.027596 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.882680 (+855084us) raft_consensus.cc:1671] finished
0426 13:19:44.882700 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.882813 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6001,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"thread_start_us":65928,"threads_started":1,"wal-append.queue_time_us":374326}
I20240426 13:19:44.941741 27323 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:44.973088 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4018963779999467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
I20240426 13:19:44.940363 25103 rpcz_store.cc:276] 0426 13:19:43.128726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.011252 (+882526us) service_pool.cc:225] Handling call
0426 13:19:44.022085 (+ 10833us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:44.246104 (+224019us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.246135 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:44.265973 (+ 19838us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.439173 (+173200us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b180016d980. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:44.781773 (+342600us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:44.790665 (+ 8892us) log.cc:842] Serialized 14 byte log entry
0426 13:19:44.796864 (+ 6199us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.796937 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.889910 (+ 92973us) raft_consensus.cc:1671] finished
0426 13:19:44.890003 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.890124 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":290138,"prepare.queue_time_us":8614,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":245712,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":10196,"spinlock_wait_cycles":3844864,"thread_start_us":342393,"threads_started":1,"wal-append.queue_time_us":545294}
I20240426 13:19:44.985805 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.978000 25027 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.979938 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.980234 25022 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:44.987565 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:44.989001 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.989737 25022 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:44.990739 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:44.992398 25022 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:44.958276 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.979504 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.982865 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:44.994354 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:44.994957 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:44.965008 27114 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:44.995707 27114 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:44.995486 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.928951 27150 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:44.966279 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:44.997444 25101 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:44.965685 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:44.973202 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:44.999519 27150 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:44.999773 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.000095 27150 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.001160 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.002177 27141 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:44.995539 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.004776 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.006403 25101 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:44.986493 25031 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:19:45.008944 27349 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20240426 13:19:45.009308 27273 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:44.975172 27263 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.011271 27339 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.018411 27253 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.020301 27323 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.021600 27323 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.025918 27303 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.026564 27303 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.978195 26945 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.028574 26945 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.029021 26945 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.034564 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 288) took 2390 ms. Trace:
I20240426 13:19:45.035066 25028 rpcz_store.cc:276] 0426 13:19:42.644238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:42.644414 (+ 176us) service_pool.cc:225] Handling call
0426 13:19:44.220382 (+1575968us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:44.249241 (+ 28859us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:45.034278 (+785037us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54000b1150. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:45.034470 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3623674624}
I20240426 13:19:44.977442 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.040717 27331 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.041185 27331 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.044346 25022 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20240426 13:19:45.044692 26965 consensus_queue.cc:1159] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 1 last_received_current_leader { term: 2 index: 2 } }
I20240426 13:19:45.045964 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 305) took 1006 ms. Trace:
I20240426 13:19:45.046471 25022 rpcz_store.cc:276] 0426 13:19:44.039053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.993398 (+954345us) service_pool.cc:225] Handling call
0426 13:19:44.993807 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:45.044914 (+ 51107us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:45.045667 (+ 753us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b54002ae754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:45.045892 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115507584}
I20240426 13:19:45.049144 25099 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:45.053315 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 67993.7us
I20240426 13:19:45.060927 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.068588 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.073057 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.073998 25102 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:45.074604 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.074810 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.075186 25029 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.076005 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 100406us
I20240426 13:19:45.077899 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.079581 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.080286 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:44.979306 24952 rpcz_store.cc:276] 0426 13:19:43.266067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.544546 (+278479us) service_pool.cc:225] Handling call
0426 13:19:43.588208 (+ 43662us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:44.034033 (+445825us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:44.034065 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:44.034533 (+ 468us) log.cc:842] Serialized 26 byte log entry
0426 13:19:44.294011 (+259478us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:44.294060 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.294119 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:44.950789 (+656670us) raft_consensus.cc:1671] finished
0426 13:19:44.950823 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:44.951024 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":445383,"raft.queue_time_us":447234,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":259038,"threads_started":1,"wal-append.queue_time_us":779292}
I20240426 13:19:45.082871 25105 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:45.083681 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 100865us
I20240426 13:19:45.081315 26824 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.085165 26824 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.085522 27350 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.085582 26824 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:44.996170 27114 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.089442 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.098517 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.091374 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.103217 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.103989 25022 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.070590 27272 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.073091 27150 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.079146 27197 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.131412 27197 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.131811 27197 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:45.132196 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54819.8us
I20240426 13:19:45.133467 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.048801 26912 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.080778 27273 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.105448 27359 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.110854 27141 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.103278 27249 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.095185 27311 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.128314 27272 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.130918 27150 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.109095 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.101935 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:45.164547 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62731.1us
I20240426 13:19:45.165441 27359 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.165808 27359 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.166132 27359 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.166297 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.135301 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.167069 27113 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.142100 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.168668 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.169369 27113 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.169518 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.169793 27113 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.170157 27113 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.170271 27310 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.172477 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.154011 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.093464 27108 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.144320 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:45.097568 27107 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.173954 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:45.235303 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126327us
I20240426 13:19:45.143810 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:45.236460 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:45.240196 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 67799.6us
W20240426 13:19:45.245815 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71905.9us
I20240426 13:19:45.282670 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.160498 26839 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.114931 26996 consensus_queue.cc:1035] T f4076b67a4fe4b3297158d094e10de08 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240426 13:19:45.287326 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.283298 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.158403 27249 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.159303 26965 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.127s
I20240426 13:19:45.286760 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.287120 25029 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.172549 27150 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.160440 27305 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.294332 25099 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.096324 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:45.295387 25098 raft_consensus.cc:595] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.288409 24996 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.171967 27311 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.290768 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.238098 27350 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.160703 27273 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.185971 27339 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.245043 27310 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.296749 24996 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.302222 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.283289 27107 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.174774 27365 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.306296 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.295156 26839 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.304968 27311 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.294203 27305 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.293206 27249 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.295307 26912 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:45.305291 27366 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.329186 27147 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.332067 27272 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.333122 27273 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.313375 27107 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:45.333945 27271 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.536s user 0.007s sys 0.010s
I20240426 13:19:45.336827 27171 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.336611 27113 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.338644 27141 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.338235 27093 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:45.314245 26656 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.002s
I20240426 13:19:45.321117 27339 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.340008 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1390) took 2154 ms. Trace:
I20240426 13:19:45.340111 27310 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:45.377617 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.366034 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.333665 27365 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.289047 27370 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20240426 13:19:45.372918 27141 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.354373 27305 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.386719 27141 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.374439 25024 rpcz_store.cc:276] 0426 13:19:43.185710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:43.188104 (+ 2394us) service_pool.cc:225] Handling call
0426 13:19:43.250141 (+ 62037us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:43.263250 (+ 13109us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:43.548462 (+285212us) log.cc:842] Serialized 14 byte log entry
0426 13:19:43.618581 (+ 70119us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:43.618644 (+ 63us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:43.618700 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:44.933375 (+1314675us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b180042b2c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59
0426 13:19:45.339786 (+406411us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:45.339838 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:45.339953 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":221623,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":112,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":67790464,"thread_start_us":339599,"threads_started":2,"wal-append.queue_time_us":61344,"wal-append.run_cpu_time_us":1076,"wal-append.run_wall_time_us":1005814}
W20240426 13:19:45.372242 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 135900us
I20240426 13:19:45.357723 27366 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.374388 25103 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.374763 25098 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.377810 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.331910 26945 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.401404 26945 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.363970 27107 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.401093 24854 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:45.404191 27366 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.404440 25100 raft_consensus.cc:2354] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:45.405783 27365 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.342810 25026 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:45.380192 25620 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:45.408073 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.380360 25105 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.409354 24920 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
W20240426 13:19:45.367276 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 233793us
I20240426 13:19:45.354586 26839 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.399647 27113 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.395195 25102 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:45.409754 24920 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240426 13:19:45.410462 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.369285 27271 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.422514 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.423544 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:45.425189 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 137951us
W20240426 13:19:45.429807 27373 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240426 13:19:45.376214 26656 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:45.413177 27369 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20240426 13:19:45.432628 27379 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.433229 26656 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.389055 25617 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:45.421285 27171 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.436426 27365 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.436856 27365 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.437207 27365 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.426026 27370 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:45.440289 27370 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:45.441404 27370 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.442247 26839 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.443025 27370 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.445478 27147 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.449613 27197 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.451529 27197 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.453430 27171 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:45.453243 27150 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.454298 27303 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.381230 27310 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.456746 27378 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 13:19:45.458074 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 167001us
I20240426 13:19:45.459932 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:45.462949 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:45.470337 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46918.7us
I20240426 13:19:45.474308 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:45.412194 26965 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:45.396471 26824 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.480074 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.481266 24947 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:19:45.482758 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60355.1us
I20240426 13:19:45.494127 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.489166 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.482443 27093 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.497398 26656 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.497944 26656 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.498934 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.500222 27171 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.500658 27171 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.501029 27171 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.506039 26824 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.508404 25026 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:45.508922 27360 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.509469 27360 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:45.509675 26656 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.509946 27360 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.511086 27366 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.513065 27271 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.513916 25617 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:45.485302 26965 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.514957 26965 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.519402 25030 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 1.
I20240426 13:19:45.520936 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:45.492264 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 195570us
I20240426 13:19:45.525949 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.498276 27379 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:45.527820 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.521718 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.565922 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.525903 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.569659 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.473403 26979 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20240426 13:19:45.573613 26824 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.496459 27253 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.574674 27150 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.615947 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.576932 27365 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.565428 27339 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20240426 13:19:45.578531 27107 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.579880 27171 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.581596 27310 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.582207 27263 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:45.526208 27176 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.585089 27311 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.631467 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.633224 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.634136 24949 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:45.588390 26679 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.635s user 0.006s sys 0.003s
I20240426 13:19:45.634900 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.635118 26679 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.636065 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:45.636663 27366 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.636852 24954 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:45.637149 26679 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.637686 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.638294 24956 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.515826 27276 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
W20240426 13:19:45.638533 27114 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.504s user 0.007s sys 0.000s
I20240426 13:19:45.638922 27114 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.589094 27303 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.640194 27311 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.566114 24996 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.598702 25022 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20240426 13:19:45.641150 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75099.4us
I20240426 13:19:45.641927 25027 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:45.611903 27271 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:45.643239 25097 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.644017 25096 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:45.625293 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:45.646600 24951 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.600126 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.493569 27197 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:45.600700 27174 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20240426 13:19:45.609406 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.604386 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1500) took 1272 ms. Trace:
W20240426 13:19:45.597251 27331 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.555s user 0.006s sys 0.006s
I20240426 13:19:45.649614 24947 rpcz_store.cc:276] 0426 13:19:44.331955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:44.673203 (+341248us) service_pool.cc:225] Handling call
0426 13:19:45.604238 (+931035us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54003683d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:45.604335 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1550343040}
W20240426 13:19:45.654597 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75468.8us
I20240426 13:19:45.654735 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.655267 24948 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.575817 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97296.3us
I20240426 13:19:45.600944 27147 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.604516 27378 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:45.664059 27331 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.664459 27378 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.665652 24851 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39848 (request call id 278) took 14168 ms (client timeout 15000 ms). Trace:
I20240426 13:19:45.665987 27331 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.667155 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.616739 27253 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.611845 26824 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.628610 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.676923 24955 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.678062 27174 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.678434 27174 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:45.678686 27360 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.679183 27174 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.630395 27176 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.680248 27176 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.682205 25030 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:45.683477 27365 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.653143 27366 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20240426 13:19:45.616497 27108 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:45.680176 25029 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.676541 26824 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.621850 26945 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20240426 13:19:45.660804 27114 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.638418 25621 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:45.654208 27276 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:45.694223 27276 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.694586 27276 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.697618 27176 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.612309 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.711673 25025 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.653214 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.712509 24952 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.712836 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:45.713495 25621 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:45.683897 27359 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.517s user 0.006s sys 0.000s
I20240426 13:19:45.714380 27359 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:45.717000 24953 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.717885 27359 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.718376 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.718945 24953 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.718839 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.720686 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.721491 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.722002 25101 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.723982 27176 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:45.656066 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 86460.1us
I20240426 13:19:45.648022 27249 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.733947 26912 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.660781 27107 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.663118 26969 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.738759 27249 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.667831 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.740845 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.741255 27360 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.666327 27378 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.671873 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:45.672513 27323 raft_consensus.cc:979] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:45.672597 25615 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240426 13:19:45.743372 27323 raft_consensus.cc:1067] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Signalling peer 1e87e7bb414642aeb72fe0d2ff72cc2e to start an election
I20240426 13:19:45.673892 27171 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:45.673700 27141 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.675252 27375 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.745819 27375 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:45.746661 27375 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.747208 27141 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.667042 24854 catalog_manager.cc:5484] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e reported cstate change: term changed from 1 to 3, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 3 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:19:45.676424 27253 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.700865 26824 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.666011 24851 rpcz_store.cc:269] 0426 13:19:31.497070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:31.508165 (+ 11095us) service_pool.cc:225] Handling call
0426 13:19:45.665463 (+14157298us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:38.538554 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:38.539011 (+ 457us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:38.539051 (+ 40us) write_op.cc:427] Acquired schema lock
0426 13:19:38.539068 (+ 17us) tablet.cc:660] Decoding operations
0426 13:19:38.552213 (+ 13145us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:38.552269 (+ 56us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:38.552294 (+ 25us) tablet.cc:683] Acquiring locks for 9 operations
0426 13:19:38.553020 (+ 726us) tablet.cc:699] Row locks acquired
0426 13:19:38.553039 (+ 19us) write_op.cc:260] PREPARE: finished
0426 13:19:38.553230 (+ 191us) write_op.cc:270] Start()
0426 13:19:38.553348 (+ 118us) write_op.cc:275] Timestamp: P: 1714137578553154 usec, L: 0
0426 13:19:38.553367 (+ 19us) op_driver.cc:331] REPLICATION: starting
0426 13:19:38.554015 (+ 648us) log.cc:842] Serialized 3639 byte log entry
0426 13:19:38.823009 (+268994us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8d3cee2e 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b
0426 13:19:39.262119 (+439110us) op_driver.cc:444] REPLICATION: finished
0426 13:19:39.542809 (+280690us) write_op.cc:300] APPLY: starting
0426 13:19:39.542896 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:39.543105 (+ 209us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:39.543129 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:39.544093 (+ 964us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:39.544295 (+ 202us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0426 13:19:39.544340 (+ 45us) write_op.cc:311] APPLY: finished
0426 13:19:39.544613 (+ 273us) log.cc:842] Serialized 71 byte log entry
0426 13:19:39.544758 (+ 145us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:39.544878 (+ 120us) write_op.cc:446] Released schema lock
0426 13:19:39.544918 (+ 40us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:39.545193 (+ 275us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:39.545221 (+ 28us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":179346,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":249717,"apply.run_cpu_time_us":2741,"apply.run_wall_time_us":2742,"num_ops":9,"prepare.queue_time_us":296811,"prepare.run_cpu_time_us":8471,"prepare.run_wall_time_us":302064,"raft.queue_time_us":274958,"raft.run_cpu_time_us":15685,"raft.run_wall_time_us":196958,"replication_time_us":663363,"spinlock_wait_cycles":3329408,"thread_start_us":693353,"threads_started":4,"wal-append.queue_time_us":313078,"wal-append.run_cpu_time_us":26946,"wal-append.run_wall_time_us":1810187}]]}
I20240426 13:19:45.753557 26824 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.702781 24947 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.703881 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.704890 27174 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.706422 27147 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.724887 26965 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.763911 26965 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.725555 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.765759 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.765933 24953 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240426 13:19:45.726780 27114 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.728727 25023 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:45.771271 26945 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:45.772965 26945 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:45.773329 26945 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.776672 25104 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.732142 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.743947 26713 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:45.734095 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.735260 25620 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:45.657619 27386 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240426 13:19:45.735270 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 78543.6us
I20240426 13:19:45.737080 24956 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:45.784448 27147 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.784750 24996 leader_election.cc:398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:45.785068 24996 leader_election.cc:401] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20240426 13:19:45.785251 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51232.8us
I20240426 13:19:45.747979 27375 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.785992 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:45.613561 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 129090us
I20240426 13:19:45.738364 24948 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:45.741369 24949 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.671284 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.673559 27305 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:45.605114 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 83495.2us
I20240426 13:19:45.761744 27380 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:19:45.722153 24952 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:45.783113 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55088.4us
I20240426 13:19:45.791853 24951 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:45.783689 26713 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.796506 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.793771 27386 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:45.803717 27176 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.804181 27305 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.797279 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.797255 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:45.804900 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.798079 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.805501 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:45.794618 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.806154 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.795357 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.806564 25026 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.774230 26945 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:45.807655 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.829972 24949 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:45.810213 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.803061 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:45.811432 27305 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:45.811950 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:45.812484 27171 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.813218 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.806773 24951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "242110e36ca34176b9d522087d440de6"
dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:41286
I20240426 13:19:45.793459 27147 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.793141 27323 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.793365 27114 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.823377 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.823763 25589 leader_election.cc:398] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5.
I20240426 13:19:45.824141 25104 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:45.824285 27029 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.825174 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.825172 27108 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:45.816496 27141 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.832531 24956 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.834718 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.837241 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.808696 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:45.775241 27276 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.839495 25025 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:45.846434 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:45.881464 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57892.8us
W20240426 13:19:45.886929 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80829.7us
I20240426 13:19:45.923949 25098 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:45.924433 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:45.887216 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92663.9us
W20240426 13:19:45.931489 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96896.4us
I20240426 13:19:45.931738 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.851974 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.932252 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.810680 24952 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:45.925396 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:45.933281 25097 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.926541 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.881655 25096 raft_consensus.cc:2362] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5. Re-sending same reply.
I20240426 13:19:45.934310 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.935075 24955 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20240426 13:19:45.936638 24996 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.937078 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.937326 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:45.927337 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90191.9us
I20240426 13:19:45.886539 25100 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:45.928903 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.950138 25096 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:45.938509 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.951438 25096 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:45.808203 26713 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:45.888116 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.938470 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.967375 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.967873 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.968814 27382 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:45.969555 26996 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:45.947327 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.970448 25069 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.970580 27271 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:45.970654 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.971089 27271 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:45.971297 25069 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.971540 27271 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:45.936724 26912 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.972074 26912 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.972456 26912 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:45.972738 27323 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:45.972895 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.973320 27323 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.973798 27377 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:45.974576 27372 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.974254 27323 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.975055 27372 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.976141 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.946621 26656 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:19:45.979585 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.979719 27393 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:45.980365 27350 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:45.946092 27113 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:45.980700 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:45.981120 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:45.955729 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:45.955848 27385 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:45.975884 27372 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.982738 27197 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:45.986102 27203 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:45.986459 27203 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.987226 27203 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.956542 27108 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:45.958508 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:45.988365 27203 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.974287 27377 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.988756 27376 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240426 13:19:45.989348 27376 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.989449 27377 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.990281 27376 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.959631 27176 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:45.952980 26965 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20240426 13:19:45.992656 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.988448 27108 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.993435 25026 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:45.993527 27360 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:45.994154 27360 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:45.994277 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.995411 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.995764 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:45.960527 27394 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20240426 13:19:45.996810 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.996666 27253 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.963119 27276 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:45.998121 25025 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:45.995972 25026 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:45.998534 27253 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:45.999133 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.000713 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.951778 27311 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:45.838743 27305 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:45.969326 27352 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:45.979151 26839 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.935865 27114 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:45.971554 27249 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:45.994010 27108 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:45.995056 27360 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.002123 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.015089 25030 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:45.885905 27386 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:45.932615 25099 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:45.961042 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:46.004431 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.000799 27107 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.005287 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.988451 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.953966 24996 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.008666 25028 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240426 13:19:46.009092 25103 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:45.935708 26824 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:45.948184 27378 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20240426 13:19:46.010027 27311 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:45.948014 26945 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.017535 24956 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:45.936298 27310 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:46.021245 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.052104 27378 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:46.052326 27263 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:46.052597 27378 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.052896 27263 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.021839 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:46.025462 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:45.963764 27150 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:46.029224 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:45.960312 27280 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:46.037556 26965 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:46.025506 25026 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:46.011682 27393 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.012023 27305 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:46.038338 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84442.7us
I20240426 13:19:46.048352 26839 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:46.041381 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 52989.9us
I20240426 13:19:46.140022 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.014513 27249 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:45.838934 27359 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.031226 25030 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:46.049201 27331 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.078985 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.000468 25621 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:46.080315 27276 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.138446 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:46.033788 27386 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:46.069630 27263 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.034550 27353 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240426 13:19:46.152760 26656 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.070475 26965 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:46.153024 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74142.9us
I20240426 13:19:46.049440 27113 raft_consensus.cc:2741] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240426 13:19:46.159421 27366 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.080312 27150 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.054646 26969 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:46.053627 27378 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.040695 27147 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.151926 27311 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.180404 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.139328 27107 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:19:46.150619 24951 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:46.206516 24951 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.144397 27249 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.190855 25022 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240426 13:19:46.160498 27382 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.200142 24920 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:46.191807 26965 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:46.199995 26824 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:46.190250 27398 log.cc:924] Time spent T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20240426 13:19:46.206079 25098 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.153285 27391 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20240426 13:19:46.191545 27350 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.150455 27310 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20240426 13:19:46.184298 27365 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:46.220158 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.184942 27271 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.230726 25026 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.223090 25029 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:46.210189 27263 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.209932 27114 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.233700 25098 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:46.219159 25069 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:46.234421 25098 raft_consensus.cc:595] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.237489 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.219518 27359 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.220762 27377 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.228121 27365 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:46.254544 25620 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240426 13:19:46.251183 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109127us
W20240426 13:19:46.263170 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72458.9us
I20240426 13:19:46.233705 27339 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:46.274632 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.293742 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.253705 27391 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:46.295588 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.305019 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:46.276687 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.260537 27107 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:46.321599 27399 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20240426 13:19:46.322185 27401 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240426 13:19:46.322343 26996 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.324595 27372 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:46.325479 27029 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:46.325990 27366 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.326853 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:46.408581 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 81809.7us
I20240426 13:19:46.327526 27400 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.328258 27404 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.269629 27365 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:46.330250 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53667.7us
I20240426 13:19:46.332726 27107 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:46.384428 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 133514us
W20240426 13:19:46.387367 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 150596us
I20240426 13:19:46.266314 27359 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.307081 27197 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:46.251645 27114 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.292672 27339 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.232378 27331 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.304023 27391 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.252933 27405 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20240426 13:19:46.334072 27276 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.430075 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.438964 24921 leader_election.cc:398] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:46.400646 27372 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.399878 27029 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:46.271307 27377 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.451700 24921 leader_election.cc:401] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:46.269177 27150 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.424976 27360 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:46.442616 27176 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20240426 13:19:46.435113 27400 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:46.448467 27174 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.436342 27366 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:46.428318 27386 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.468864 27107 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.477814 26713 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.502s user 0.009s sys 0.000s
I20240426 13:19:46.478765 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:46.400915 27350 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.460975 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.307833 26824 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.410995 26996 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.489454 25620 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:19:46.462388 27197 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.527873 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.486188 27171 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6.
I20240426 13:19:46.499042 27366 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:46.533941 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 103979us
I20240426 13:19:46.538924 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:46.541626 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132322us
I20240426 13:19:46.467968 26912 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.550489 25070 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.531648 27310 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:46.491086 27305 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.561724 27310 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.576370 27310 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.546792 27171 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:46.580338 27400 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.581418 27414 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240426 13:19:46.581192 27420 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20240426 13:19:46.537927 27379 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.533890 27271 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.540993 27366 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.498893 27174 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.533704 27386 raft_consensus.cc:2741] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240426 13:19:46.561866 26839 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240426 13:19:46.584973 25026 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:46.642203 24955 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:46.570062 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.571532 24951 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.533027 27360 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:46.527781 26713 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.583128 27108 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:46.562625 27107 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.643743 24956 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:46.572294 27402 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.319s user 0.001s sys 0.000s
I20240426 13:19:46.583348 27253 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:46.654093 25024 raft_consensus.cc:2354] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
W20240426 13:19:46.658000 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106536us
I20240426 13:19:46.580371 27411 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:46.676717 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.641420 27276 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.682469 25102 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.683275 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.683336 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.685086 25103 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:46.643776 27409 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:46.536015 27391 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.689899 27114 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:46.690660 27391 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:46.692999 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.693770 25100 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.694233 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.595758 27303 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20240426 13:19:46.557822 27359 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.656081 27385 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.700592 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.700286 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.661429 27386 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.601934 26965 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.704908 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44823.7us
I20240426 13:19:46.558284 27029 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.583176 27372 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.709255 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.738226 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.738978 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.739145 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.739768 25101 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:46.739730 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:46.742632 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:46.743616 24949 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20240426 13:19:46.744649 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.745524 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.745746 25101 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:46.745981 27174 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.746315 25022 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.747092 25101 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:46.741844 27403 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20240426 13:19:46.702587 27409 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.750064 26656 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:19:46.750623 27409 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:46.748870 26713 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:46.753356 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.753360 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.756320 25030 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:46.754030 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.756623 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.754881 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.757210 25022 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.703236 27108 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:46.763092 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.760564 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.688304 27276 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:46.766782 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.767371 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:46.768057 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.740451 27377 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:46.767781 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102933us
I20240426 13:19:46.774390 25029 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:46.759665 27407 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.308s user 0.001s sys 0.001s
I20240426 13:19:46.758123 26839 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:46.708190 27386 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.692221 27428 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.208s
I20240426 13:19:46.763468 27271 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240426 13:19:46.801355 26965 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.705538 27385 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.804329 27435 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:46.840893 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.785548 26656 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:46.808445 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 312) took 1003 ms. Trace:
I20240426 13:19:46.845223 25024 rpcz_store.cc:276] 0426 13:19:45.805262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.805394 (+ 132us) service_pool.cc:225] Handling call
0426 13:19:46.808184 (+1002790us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54002fab54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:46.808368 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1767372544}
I20240426 13:19:46.677875 27197 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.809859 26945 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.755s user 0.003s sys 0.006s
I20240426 13:19:46.736286 27114 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240426 13:19:46.812971 27310 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.814848 27305 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:46.816416 27276 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:19:46.818485 27425 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
I20240426 13:19:46.818598 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:46.819260 27423 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
W20240426 13:19:46.819653 27263 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.571s user 0.008s sys 0.001s
I20240426 13:19:46.822573 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.776021 27439 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.823267 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:46.823355 27419 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.222s
I20240426 13:19:46.684410 27411 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:46.815979 24951 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.775346 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:46.834971 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 265045us
I20240426 13:19:46.805657 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:46.797916 27377 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:46.799273 26996 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.797299 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:46.828955 26912 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:46.862954 25022 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:46.863174 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102646us
I20240426 13:19:46.866156 25098 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:19:46.839643 26839 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:46.968575 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:46.945035 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41286 (request call id 1813) took 1137 ms. Trace:
I20240426 13:19:46.970256 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.905211 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.842613 27271 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:46.966324 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.755725 27360 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:46.910809 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 168256us
I20240426 13:19:46.909351 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:46.954155 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.972941 24949 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:46.913283 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:46.847745 27435 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:46.932099 27422 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:46.903846 25589 leader_election.cc:398] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:19:46.951797 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.018246 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:46.971088 25022 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:46.975641 24947 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:46.926832 27310 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.020081 25589 leader_election.cc:401] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:46.977272 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:46.978555 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.980965 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:47.023783 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 45324.4us
I20240426 13:19:47.025468 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.027469 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.028355 24955 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:46.968919 27377 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:46.969971 24951 rpcz_store.cc:276] 0426 13:19:45.805941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.806669 (+ 728us) service_pool.cc:225] Handling call
0426 13:19:46.814991 (+1008322us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400367ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9030ef37 00007fda9525bb50 00007fda8e949754 00007fda8e9496e2 00007fda8e949611 00007fda8e94958c 00007fda8e948842 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:46.934530 (+119539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724330880}
I20240426 13:19:46.963351 27386 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:46.942282 27415 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
W20240426 13:19:46.943158 27436 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20240426 13:19:47.020689 25030 raft_consensus.cc:2354] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:46.947659 27311 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.022802 25024 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:46.961555 27385 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:46.923660 27249 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.703s user 0.009s sys 0.000s
I20240426 13:19:46.975777 27271 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:46.952214 27411 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:47.037707 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:46.976334 27439 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
W20240426 13:19:47.040560 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63405us
I20240426 13:19:47.039499 27377 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.042814 24954 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:47.043318 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.017424 26656 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:46.906213 27108 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.052410 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:46.981783 25617 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:47.063724 27445 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:47.065423 27435 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:47.065851 26969 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.862s user 0.004s sys 0.005s
I20240426 13:19:46.984400 26713 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20240426 13:19:47.066943 27447 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:47.069334 27365 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.625s user 0.005s sys 0.005s
W20240426 13:19:47.037498 27423 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.069s user 0.004s sys 0.000s
I20240426 13:19:47.037132 27276 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:47.074986 27171 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:47.040138 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 136476us
I20240426 13:19:47.088064 24955 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:47.043265 27422 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:47.099431 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56199.2us
I20240426 13:19:47.101967 25024 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240426 13:19:47.104508 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 66919.1us
I20240426 13:19:47.126163 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.117163 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:47.126971 25104 raft_consensus.cc:595] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:47.112342 27359 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:47.113368 27386 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:47.117702 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.121824 25105 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.127305 25104 raft_consensus.cc:603] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:47.118664 24920 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:47.128445 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:47.100508 27439 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.131471 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 377) took 1076 ms. Trace:
I20240426 13:19:47.134493 24952 rpcz_store.cc:276] 0426 13:19:46.055359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.055654 (+ 295us) service_pool.cc:225] Handling call
0426 13:19:46.056064 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.056560 (+ 496us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:46.056603 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.080131 (+ 23528us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.266900 (+186769us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:46.266944 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.267015 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.131269 (+864254us) raft_consensus.cc:1671] finished
0426 13:19:47.131297 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.131417 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":17580,"thread_start_us":79367,"threads_started":1,"wal-append.queue_time_us":491513}
I20240426 13:19:47.115916 27259 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:47.115581 26969 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.115983 27447 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:47.077143 27249 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.137707 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.146929 24955 raft_consensus.cc:2354] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:47.141356 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.138126 25031 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.139252 24921 leader_election.cc:398] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:47.184747 24921 leader_election.cc:401] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240426 13:19:47.119451 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:47.138135 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:47.185267 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46173.2us
W20240426 13:19:47.185657 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66276.8us
I20240426 13:19:47.188313 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.188903 25102 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.189188 25102 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:47.189188 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.117354 27365 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.143412 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.143455 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:47.138962 27113 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:47.144510 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.144623 24952 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:47.122926 27171 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.129223 27276 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:47.232522 25624 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:47.159406 24947 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20240426 13:19:47.172995 24949 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:47.142799 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.190115 25615 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.190546 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.141894 27400 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:47.235002 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.194188 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.235606 26824 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:47.235034 27451 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20240426 13:19:47.236305 27412 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.235641 27400 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:47.236836 27412 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.216722 27310 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:47.186820 27409 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 2. Current term is 3.
I20240426 13:19:47.229534 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.237426 27400 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:47.252007 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.252663 25623 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.235195 26713 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:47.219288 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 319) took 1057 ms. Trace:
I20240426 13:19:47.194737 27113 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:47.219836 27449 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:47.240016 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.235811 25024 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.147294 27447 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.247367 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1571) took 1083 ms. Trace:
I20240426 13:19:47.278474 24949 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:47.279202 24947 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:47.282296 24955 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:47.246903 27310 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:47.237573 27171 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.256621 25617 raft_consensus.cc:2396] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240426 13:19:47.462451 27276 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:47.470973 27382 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:47.443267 27456 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20240426 13:19:47.471514 27276 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.473466 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:47.474273 25029 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:47.474428 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.475566 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.475910 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.476343 24952 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:47.477092 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.477500 27310 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:47.477803 24956 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:47.478206 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.478529 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.478914 24952 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:47.480285 27412 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:47.481094 27366 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.839s user 0.011s sys 0.000s
I20240426 13:19:47.481619 27276 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:47.481469 27366 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.482784 24952 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:47.483402 27366 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:47.483808 24952 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:47.486204 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.487187 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.441879 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.488298 25027 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:47.442437 27360 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20240426 13:19:47.489818 27360 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:47.490156 27360 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:47.367566 27259 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8.
I20240426 13:19:47.326668 27386 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20240426 13:19:47.468675 27171 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.275610 25099 rpcz_store.cc:276] 0426 13:19:46.163933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.164248 (+ 315us) service_pool.cc:225] Handling call
0426 13:19:46.164635 (+ 387us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.190015 (+ 25380us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:46.206223 (+ 16208us) log.cc:842] Serialized 14 byte log entry
0426 13:19:46.313829 (+107606us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b180037f6c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:19:46.453388 (+139559us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.465514 (+ 12126us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.465665 (+ 151us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:46.465712 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.465802 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.196021 (+730219us) raft_consensus.cc:1671] finished
0426 13:19:47.247176 (+ 51155us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.247311 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60848,"prepare.queue_time_us":16200,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":49980,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":17211,"spinlock_wait_cycles":4250880,"thread_start_us":124568,"threads_started":1,"wal-append.queue_time_us":333569}
I20240426 13:19:47.499595 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.328370 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:47.501613 25024 raft_consensus.cc:2354] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:47.386093 25023 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:47.294009 27439 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.470005 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.297230 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:47.282649 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.503402 25624 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:47.467248 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 278133us
I20240426 13:19:47.505594 24921 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.493949 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.506906 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.507037 24954 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.258136 25104 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:47.514029 25104 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:47.514669 25104 raft_consensus.cc:595] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:47.522223 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:47.284412 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 156779us
I20240426 13:19:47.284170 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:47.571482 24947 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:47.255259 25028 rpcz_store.cc:276] 0426 13:19:46.161452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.170502 (+ 9050us) service_pool.cc:225] Handling call
0426 13:19:46.187093 (+ 16591us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:46.424204 (+237111us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:46.424250 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:46.451999 (+ 27749us) log.cc:842] Serialized 26 byte log entry
0426 13:19:46.640585 (+188586us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:46.662069 (+ 21484us) log.cc:842] Serialized 14 byte log entry
0426 13:19:46.662277 (+ 208us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:46.662346 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:47.219007 (+556661us) raft_consensus.cc:1671] finished
0426 13:19:47.219034 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:47.219216 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25178,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4201,"raft.queue_time_us":138396,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":6379,"thread_start_us":375738,"threads_started":2,"wal-append.queue_time_us":287791}
I20240426 13:19:47.586484 24952 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:47.247311 26839 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:47.588943 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66809.6us
W20240426 13:19:47.600641 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 107889us
I20240426 13:19:47.256935 26824 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:47.442492 27446 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.270s user 0.000s sys 0.000s
W20240426 13:19:47.444218 27454 log.cc:924] Time spent T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20240426 13:19:47.274245 27385 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:47.275380 27113 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:47.428926 27448 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
I20240426 13:19:47.471544 27382 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.605326 25104 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:47.603878 24995 leader_election.cc:398] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7.
I20240426 13:19:47.606719 25096 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:47.292286 27350 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.611050 25024 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:47.423394 26679 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.414131 27422 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.595892 27249 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:47.274549 27271 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.587005 27409 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.596966 27445 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:47.640084 25024 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.640504 25024 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:47.641571 25096 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20240426 13:19:47.281409 27449 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:47.602432 27400 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:47.602198 27440 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.622009 25622 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:47.642313 25096 pending_rounds.cc:76] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:19:47.642105 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240426 13:19:47.641410 25024 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.654855 24953 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:47.479914 27447 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.635123 27365 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.604586 26839 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.650171 25096 pending_rounds.cc:97] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240426 13:19:47.651985 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240426 13:19:47.644742 27377 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.544s user 0.007s sys 0.001s
I20240426 13:19:47.669526 25096 raft_consensus.cc:2866] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.645987 27449 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:47.689270 25623 raft_consensus.cc:1226] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.631036 27385 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:47.689841 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92167.1us
I20240426 13:19:47.634394 26679 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.721588 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:47.613832 26824 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.646521 27391 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20240426 13:19:47.633078 27350 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.721478 24956 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:47.636958 27422 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.732986 27462 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20240426 13:19:47.754952 25101 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:47.761634 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 158254us
I20240426 13:19:47.658075 27416 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
W20240426 13:19:47.730650 27359 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.587s user 0.005s sys 0.000s
I20240426 13:19:47.791667 27359 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:47.658526 27445 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20240426 13:19:47.796226 27339 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.804316 25097 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:19:47.754256 27271 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:47.771492 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 160190us
I20240426 13:19:47.658849 27440 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.762862 25099 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:47.733557 27107 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.065s user 0.012s sys 0.000s
I20240426 13:19:47.778699 25616 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:47.638602 27360 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.782931 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.823246 25096 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:47.688143 27377 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.817915 27400 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240426 13:19:47.732815 24954 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:47.815757 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 182110us
W20240426 13:19:47.816381 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 176939us
W20240426 13:19:47.824082 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 101989us
I20240426 13:19:47.823004 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:47.849267 25026 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:19:47.849653 25026 pending_rounds.cc:76] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:47.849880 25026 pending_rounds.cc:97] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:19:47.850104 25026 raft_consensus.cc:2866] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.851837 26679 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.853405 26656 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.752s user 0.010s sys 0.000s
I20240426 13:19:47.853683 26824 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.860496 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 67928.3us
I20240426 13:19:47.843394 27327 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.853833 27479 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:47.879109 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49846.6us
I20240426 13:19:47.843909 27409 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.847719 27271 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.853849 26656 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:47.918846 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:47.919852 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:47.913522 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83078us
I20240426 13:19:47.926323 25104 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240426 13:19:47.926498 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63896.9us
I20240426 13:19:47.880941 27310 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.934515 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:47.932317 24995 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:47.886157 27327 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.912649 27476 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.932622 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.945520 25103 raft_consensus.cc:2396] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 5 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240426 13:19:47.945936 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1404) took 1270 ms. Trace:
I20240426 13:19:47.946218 25103 rpcz_store.cc:276] 0426 13:19:46.675709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.924864 (+1249155us) service_pool.cc:225] Handling call
0426 13:19:47.945885 (+ 21021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:47.950054 27376 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.950767 27376 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.954334 25026 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 1. Current term is 2. Ops: []
I20240426 13:19:47.900388 27409 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.939827 25098 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1572) took 1488 ms (client timeout 1570 ms). Trace:
I20240426 13:19:47.941566 25104 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240426 13:19:47.958786 25104 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:19:47.959570 27485 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.930706 26839 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.904101 27271 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.930967 27447 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:47.960776 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1817) took 1957 ms (client timeout 2207 ms). Trace:
I20240426 13:19:47.961863 25026 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:47.972618 25026 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:47.972366 24950 rpcz_store.cc:269] 0426 13:19:46.002900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.003377 (+ 477us) service_pool.cc:225] Handling call
0426 13:19:47.960519 (+1957142us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540038e5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.960685 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2440853888}
I20240426 13:19:47.963764 27485 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.959187 25104 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
W20240426 13:19:47.958573 25098 rpcz_store.cc:269] 0426 13:19:46.451076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.451235 (+ 159us) service_pool.cc:225] Handling call
0426 13:19:47.939580 (+1488345us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.939748 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":774660992}
I20240426 13:19:47.979717 25104 raft_consensus.cc:2894] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
I20240426 13:19:47.983242 25104 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:47.967506 27410 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.894114 27479 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:47.993016 27487 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:47.995551 27487 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:47.997233 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1901) took 1991 ms (client timeout 2004 ms). Trace:
W20240426 13:19:47.997691 25023 rpcz_store.cc:269] 0426 13:19:46.006174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.006330 (+ 156us) service_pool.cc:225] Handling call
0426 13:19:47.997070 (+1990740us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000b0754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.997179 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2524216704}
I20240426 13:19:47.999644 25029 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:47.959945 27382 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:47.987874 25026 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:47.975071 27385 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.997267 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1416) took 1297 ms (client timeout 1644 ms). Trace:
W20240426 13:19:48.004228 25027 rpcz_store.cc:269] 0426 13:19:46.700222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.700363 (+ 141us) service_pool.cc:225] Handling call
0426 13:19:47.997111 (+1296748us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:47.997222 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1809761024}
I20240426 13:19:48.006111 27422 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.006196 27482 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20240426 13:19:48.006937 27271 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.009966 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:47.981786 27350 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:47.996773 27477 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240426 13:19:47.991868 26839 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:47.992106 27113 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.014655 27377 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.177999 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1517) took 1017 ms. Trace:
I20240426 13:19:48.003087 26824 raft_consensus.cc:979] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
W20240426 13:19:48.188834 27484 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20240426 13:19:48.006939 27485 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.016817 25026 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.098532 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.130702 25096 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:48.188632 24955 rpcz_store.cc:276] 0426 13:19:47.160175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.160341 (+ 166us) service_pool.cc:225] Handling call
0426 13:19:48.177774 (+1017433us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54003683d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.177932 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106162560}
I20240426 13:19:48.275151 27491 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20240426 13:19:47.961301 27476 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.278656 27476 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.059891 27350 raft_consensus.cc:2741] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20240426 13:19:48.280508 27479 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.047384 27493 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.267s
I20240426 13:19:48.281155 27271 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:48.279945 25026 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:48.282241 27271 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.282699 27271 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.284790 27310 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.286626 27376 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:48.287678 27311 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.233s user 0.011s sys 0.000s
I20240426 13:19:48.288292 25100 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:48.288791 27409 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.290192 27449 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.578s user 0.007s sys 0.001s
I20240426 13:19:48.290643 27449 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:48.292189 27460 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.283s user 0.001s sys 0.001s
I20240426 13:19:48.293977 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1916) took 1183 ms. Trace:
I20240426 13:19:48.294361 25022 rpcz_store.cc:276] 0426 13:19:47.110094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.111613 (+ 1519us) service_pool.cc:225] Handling call
0426 13:19:47.111944 (+ 331us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.120869 (+ 8925us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.120916 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.125753 (+ 4837us) log.cc:842] Serialized 26 byte log entry
0426 13:19:47.327134 (+201381us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:47.669014 (+341880us) log.cc:842] Serialized 14 byte log entry
0426 13:19:47.680567 (+ 11553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:47.680665 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:48.293712 (+613047us) raft_consensus.cc:1671] finished
0426 13:19:48.293742 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:48.293909 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15064,"prepare.queue_time_us":453050,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":22395,"raft.queue_time_us":190033,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":11696,"spinlock_wait_cycles":6908800,"thread_start_us":441710,"threads_started":2,"wal-append.queue_time_us":540903}
W20240426 13:19:48.297905 27171 consensus_meta.cc:287] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.701s user 0.009s sys 0.000s
I20240426 13:19:48.299319 27391 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.300122 27391 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:48.300496 27391 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.301447 27391 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.186738 27486 raft_consensus.cc:979] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: : Instructing follower 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:48.176371 27339 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:48.306677 24996 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:48.174964 27327 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.319289 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.326148 24947 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.011s sys 0.033s
W20240426 13:19:48.328500 24947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.012s sys 0.033s
I20240426 13:19:48.328696 25104 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:48.332407 24948 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:48.302991 27391 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:48.102155 27495 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:48.104168 27440 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.284827 27487 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.365794 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.009s sys 0.027s
W20240426 13:19:48.391413 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s user 0.009s sys 0.027s
I20240426 13:19:48.438408 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.102850 27461 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.346s
I20240426 13:19:48.440261 25622 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:48.437733 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.018s sys 0.033s
W20240426 13:19:48.461055 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.041s sys 0.047s
I20240426 13:19:48.007946 27410 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:48.462754 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s user 0.019s sys 0.034s
W20240426 13:19:48.547757 24981 heartbeater.cc:644] Failed to heartbeat to 127.23.61.62:34153 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.23.61.62:34153 timed out after 15.000s (SENT)
W20240426 13:19:48.466403 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.035s sys 0.039s
I20240426 13:19:48.546418 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.546195 27406 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.553886 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1565) took 2701 ms. Trace:
W20240426 13:19:48.462821 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s user 0.041s sys 0.047s
I20240426 13:19:48.561342 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.595407 25097 rpcz_store.cc:276] 0426 13:19:45.852133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.948867 (+ 96734us) service_pool.cc:225] Handling call
0426 13:19:46.012474 (+ 63607us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.804808 (+1792334us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.553561 (+748753us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.553815 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4120238336}
I20240426 13:19:48.339776 27377 raft_consensus.cc:1067] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
W20240426 13:19:48.346223 27416 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.026s sys 0.021s
I20240426 13:19:48.583815 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:48.586963 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s user 0.047s sys 0.060s
W20240426 13:19:48.606496 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 735238us
I20240426 13:19:48.534396 27492 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.574860 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.346189 27350 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.437994 27432 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:48.466481 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.043s sys 0.043s
I20240426 13:19:48.101998 27490 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.432s
I20240426 13:19:48.340699 27495 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:48.603642 27429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.001s sys 0.016s
I20240426 13:19:48.437355 27489 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.547060 27474 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:48.368782 27376 raft_consensus.cc:1067] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Signalling peer 46106ff992984d33bb4a4d8f30162639 to start an election
I20240426 13:19:48.605439 24950 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240426 13:19:48.587109 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.035s sys 0.073s
W20240426 13:19:48.626830 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.035s sys 0.073s
I20240426 13:19:48.607501 24956 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:48.627123 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 690362us
I20240426 13:19:48.608583 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:48.547266 27481 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.566737 27378 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:48.548611 27113 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.610428 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:48.576900 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:48.631635 25620 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.631875 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.606513 27499 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.334311 26839 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.508370 27487 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.624342 24921 leader_election.cc:398] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:48.625058 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.609668 26945 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.604346 25616 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
W20240426 13:19:48.612622 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.992s user 0.044s sys 0.052s
I20240426 13:19:48.641731 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:48.666569 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.05685e+06us
I20240426 13:19:48.667083 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.614037 27479 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:48.631623 27429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.634s user 0.001s sys 0.016s
I20240426 13:19:48.628758 27311 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.616233 27406 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.618152 27432 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.664232 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.626297 27271 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.619411 27449 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.668017 27498 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.617735 27350 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:48.673830 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.053s sys 0.153s
I20240426 13:19:48.676793 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:48.679570 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.053s sys 0.154s
I20240426 13:19:48.680285 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.675348 27487 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.686909 27487 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.653282 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.689383 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.714509 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:48.715787 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.659336 27474 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.689638 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.690665 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.691237 25023 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:48.723824 25027 raft_consensus.cc:2379] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.672012 27113 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.652627 27504 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.723618 25026 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:48.678481 27376 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.733466 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:48.714359 27490 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.671391 26945 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.747225 26945 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.748871 27029 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:48.749423 27432 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.753253 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1559) took 2921 ms. Trace:
I20240426 13:19:48.753733 25105 rpcz_store.cc:276] 0426 13:19:45.831641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:45.831771 (+ 130us) service_pool.cc:225] Handling call
0426 13:19:45.925070 (+ 93299us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.133906 (+1208836us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.752935 (+1619029us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540009d8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.753184 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2751953792}
I20240426 13:19:48.729579 25621 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:48.724613 27271 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:48.757548 26839 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.654312 27499 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.758692 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1581) took 1266 ms. Trace:
I20240426 13:19:48.759222 25099 rpcz_store.cc:276] 0426 13:19:47.492286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.499327 (+ 7041us) service_pool.cc:225] Handling call
0426 13:19:48.758474 (+1259147us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.758629 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":603227392}
I20240426 13:19:48.669052 27377 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.626039 27492 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.720500 27491 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:48.635039 27489 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:48.737238 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.041s sys 0.136s
W20240426 13:19:48.778059 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.801s user 0.041s sys 0.137s
I20240426 13:19:48.761621 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:48.778594 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 853118us
I20240426 13:19:48.779960 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.725945 27449 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.726168 27496 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.678432 27311 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.781374 27449 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.781754 27496 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.782888 27311 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.784276 27496 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:48.781376 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102272us
I20240426 13:19:48.734622 27504 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.676324 27479 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.787997 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.758500 27497 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.790992 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:48.797441 25022 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240426 13:19:48.668777 27481 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.799821 27481 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.801452 25028 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:48.801821 25028 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:48.802461 25028 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.803544 25028 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:48.798174 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 569) took 2779 ms. Trace:
I20240426 13:19:48.804816 25615 rpcz_store.cc:276] 0426 13:19:46.010789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.010950 (+ 161us) service_pool.cc:225] Handling call
0426 13:19:46.025088 (+ 14138us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.234343 (+1209255us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.790184 (+1555841us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.790333 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2679076736}
I20240426 13:19:48.744840 27480 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20240426 13:19:48.808821 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 379) took 1516 ms. Trace:
I20240426 13:19:48.809160 25623 rpcz_store.cc:276] 0426 13:19:47.292486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.292608 (+ 122us) service_pool.cc:225] Handling call
0426 13:19:47.445977 (+153369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.690521 (+244544us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.808616 (+1118095us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540058e654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:48.808774 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558836480}
I20240426 13:19:48.735946 27171 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:48.797998 24954 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:48.811390 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1531) took 1368 ms (client timeout 1588 ms). Trace:
W20240426 13:19:48.814795 24948 rpcz_store.cc:269] 0426 13:19:47.442984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.475362 (+ 32378us) service_pool.cc:225] Handling call
0426 13:19:48.811222 (+1335860us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:48.811331 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1958315776}
I20240426 13:19:48.796244 25069 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:48.733568 27113 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.821743 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:48.824472 25022 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:48.859385 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:48.790522 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.013s sys 0.011s
I20240426 13:19:48.816632 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:48.844931 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 48781.4us
W20240426 13:19:48.865545 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.982s user 0.018s sys 0.011s
I20240426 13:19:48.846285 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.866792 25027 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:48.672861 27495 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.783859 27449 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:48.790458 26839 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:48.737896 27376 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.900252 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.725620 27498 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.908366 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.908902 25027 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.910110 27498 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.904786 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.905468 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46184.2us
I20240426 13:19:48.905851 25022 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:19:48.910835 25022 pending_rounds.cc:76] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:48.911067 25022 pending_rounds.cc:97] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:19:48.911067 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.911652 25022 raft_consensus.cc:2866] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:19:48.913229 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 341884us
I20240426 13:19:48.913913 25069 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:48.914197 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 97663.7us
I20240426 13:19:48.916303 24995 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:48.917551 25623 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:48.917974 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.918547 25103 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:48.918929 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.920621 25069 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.920846 25027 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:48.769944 27490 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.922885 27461 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.923710 27490 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.923530 27461 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.924527 27490 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.925464 27510 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:48.869316 27378 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.930167 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.930768 25097 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:48.876613 27391 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.878643 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.927742 25022 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:48.915597 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 248465us
I20240426 13:19:48.876734 27323 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:48.928632 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 237984us
I20240426 13:19:48.917490 27498 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:48.932813 27502 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.933291 27500 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:48.779873 27491 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:48.912515 26713 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20240426 13:19:48.929441 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.926110 27461 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.906535 27481 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.906755 26839 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.777290 27377 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.930191 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:48.943852 25022 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:48.944766 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:48.946580 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.555s (SENT) [suppressed 11 similar messages]
I20240426 13:19:48.966765 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.928123 27487 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:48.931712 27150 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:48.951627 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.838204 27410 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.950847 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.871023 27480 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20240426 13:19:48.967536 25022 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 2: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:48.943351 27323 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.970336 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:48.987895 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:48.988404 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:48.952492 27508 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:48.976384 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.994451 25098 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:48.995548 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.979439 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.982476 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.789498 27497 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:48.999061 27497 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.912837 27433 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20240426 13:19:48.966487 26839 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:48.999743 27433 consensus_queue.cc:1225] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:48.958729 27504 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:48.965914 25617 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:48.967730 27500 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:48.989252 25022 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:49.014976 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:49.015329 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 99094.2us
I20240426 13:19:48.978399 27406 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:48.969058 27350 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.011528 27492 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.025617 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:48.871377 27474 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.020334 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.031080 25097 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:49.032183 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:48.994311 27323 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:48.968554 26945 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240426 13:19:48.987530 27461 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.016177 24995 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.026454 25026 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:49.027122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 106579us
I20240426 13:19:49.008642 27496 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:48.968886 27481 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:49.027303 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 80749.4us
I20240426 13:19:48.993093 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:49.043663 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50681.2us
I20240426 13:19:49.009869 27150 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:49.039731 27461 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.051146 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.016410 27500 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:49.013222 27513 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:48.990329 27515 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:49.058892 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:49.061787 25025 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:49.009452 27517 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:49.054538 27527 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:49.051791 26973 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:49.062567 26969 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.062939 26973 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:49.063117 27527 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:49.063413 26973 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:49.063071 26969 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.055195 27386 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:48.972680 27502 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.012425 27514 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.068905 27386 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.069072 27514 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.070369 27502 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.071310 27386 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.071391 27514 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.049448 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.080847 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.081530 25097 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:49.084347 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.088856 27489 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.057580 27479 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.093979 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.094120 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:49.094854 27489 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.096519 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.097123 25097 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:49.095247 27489 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.099452 27489 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:49.099512 27491 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.099769 27481 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.098031 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:49.085225 27480 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240426 13:19:49.058499 26713 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20240426 13:19:49.109566 25026 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:49.050361 27496 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.087167 27113 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:49.110817 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51996us
I20240426 13:19:49.060084 27406 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:49.129385 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.130554 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.131278 25026 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:49.130941 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:49.132539 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.136749 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.098114 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.093223 27376 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:49.056703 27350 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20240426 13:19:49.120213 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:49.112252 27499 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:49.113870 27479 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:49.153441 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.123728 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.125303 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:49.143353 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 128705us
I20240426 13:19:49.146347 24996 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:49.147889 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.168889 25097 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:49.148623 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.174412 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.122812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.117045 27447 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.162240 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.177700 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.178725 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.178897 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.171144 25026 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:49.165868 25589 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.180830 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.183930 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.186244 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:49.190666 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.182559 27029 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:49.192220 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.192793 25023 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:49.193665 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:49.194778 25619 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:49.194871 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.195493 25023 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:19:49.195742 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.196311 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.010154 27508 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.196921 25023 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:19:49.110532 27481 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20240426 13:19:49.195979 27511 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
I20240426 13:19:49.215600 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.135049 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.199123 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.216545 25026 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:49.151007 27504 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.151796 27491 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.206861 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:49.208230 24948 pending_rounds.cc:76] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:49.209460 25623 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240426 13:19:49.155493 27377 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.149991 27113 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.234694 27375 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:49.234588 27113 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.214164 27108 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.094888 27150 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.218796 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:49.221582 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:49.213471 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.214354 27029 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:49.169948 27400 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:49.226724 24948 pending_rounds.cc:97] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:49.179695 27447 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.231899 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.196511 27472 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.189564 27439 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:49.230485 25623 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 3
I20240426 13:19:49.232821 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.305907 25623 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.4
I20240426 13:19:49.306361 25623 raft_consensus.cc:2894] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted by new leader
W20240426 13:19:49.161151 27339 consensus_meta.cc:287] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.828s user 0.008s sys 0.002s
I20240426 13:19:49.232386 25022 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
W20240426 13:19:49.233870 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 172012us
I20240426 13:19:49.206058 27461 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.197542 27479 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:49.197044 27376 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:49.214517 27249 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:49.235109 27375 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.234606 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.245110 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:49.248973 25026 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:49.248481 25028 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:49.251120 24948 raft_consensus.cc:2866] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:49.251315 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:49.251621 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:49.311297 27518 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:49.228530 27504 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.244666 27108 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.233538 27491 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.311564 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:49.724879 27489 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.625s user 0.015s sys 0.002s
I20240426 13:19:49.766384 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.758545 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.771018 27489 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.772163 27108 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:49.772063 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.772634 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.773651 25105 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:49.722714 24947 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:49.253546 27447 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:49.713008 27498 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.755s user 0.009s sys 0.000s
I20240426 13:19:49.775422 27498 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:49.773620 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.807754 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:49.806277 25099 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:49.779803 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1434) took 2588 ms. Trace:
I20240426 13:19:49.725091 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.813813 25026 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:49.334086 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:49.325438 25623 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:49.773639 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.817117 25028 rpcz_store.cc:276] 0426 13:19:47.191572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.574197 (+382625us) service_pool.cc:225] Handling call
0426 13:19:48.555686 (+981489us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b5800002a98. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:48.555981 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:48.801364 (+245383us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000f30a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:19:49.779554 (+978190us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:49.779746 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279718912}
I20240426 13:19:49.779007 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:49.722890 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 325580us
I20240426 13:19:49.818636 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.819509 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:49.327428 25101 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240426 13:19:49.762609 25615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.000s sys 0.010s
W20240426 13:19:49.326956 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 70938.5us
W20240426 13:19:49.725585 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 398485us
I20240426 13:19:49.820966 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.329931 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1585) took 1023 ms. Trace:
I20240426 13:19:49.822434 25096 rpcz_store.cc:276] 0426 13:19:48.306439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.306955 (+ 516us) service_pool.cc:225] Handling call
0426 13:19:48.307377 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:48.342930 (+ 35553us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:48.342970 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:48.343524 (+ 554us) log.cc:842] Serialized 26 byte log entry
0426 13:19:48.683245 (+339721us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:48.683279 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.683341 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:49.329673 (+646332us) raft_consensus.cc:1671] finished
0426 13:19:49.329699 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:49.329862 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158248,"raft.queue_time_us":38608,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":58630,"thread_start_us":339402,"threads_started":1,"wal-append.queue_time_us":443155}
W20240426 13:19:49.824769 24955 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.000s sys 0.014s
W20240426 13:19:49.555997 27323 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.521s user 0.008s sys 0.001s
I20240426 13:19:49.252925 27497 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:49.487362 27506 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.770s user 0.005s sys 0.000s
W20240426 13:19:49.709823 26839 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.710s user 0.011s sys 0.000s
I20240426 13:19:49.400072 27524 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:49.810868 27276 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.822017 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:49.399390 27521 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.293s user 0.000s sys 0.001s
I20240426 13:19:49.820129 27331 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:49.825065 24955 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.000s sys 0.014s
I20240426 13:19:49.833947 26945 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.255275 27400 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:49.850106 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.733376 27483 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:49.814797 27505 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:19:49.672065 27536 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.780355 27249 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.812196 27375 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:49.327833 25102 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.431254 27439 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.857791 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:49.602424 27508 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:49.859613 25105 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240426 13:19:49.820986 25615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s user 0.000s sys 0.011s
I20240426 13:19:49.887236 27475 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:49.671411 27540 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:49.775127 27479 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.326934 27514 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.547s
I20240426 13:19:49.368206 27502 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.493s
W20240426 13:19:49.882094 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60134.4us
I20240426 13:19:50.003721 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.807264 27518 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:49.883683 27524 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:49.884222 25618 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:49.884503 27276 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.862715 26839 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.885836 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1921) took 2641 ms. Trace:
I20240426 13:19:49.823608 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:49.355244 27433 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.508s
I20240426 13:19:49.355443 27515 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.526s
W20240426 13:19:49.897169 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 645587us
I20240426 13:19:49.856976 27497 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.897652 25028 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:49.459384 26973 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.815594 27376 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:49.918640 25023 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:50.013494 27518 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:49.828668 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.002s sys 0.011s
W20240426 13:19:50.048138 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s user 0.002s sys 0.011s
W20240426 13:19:49.944360 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.028s sys 0.057s
I20240426 13:19:49.535872 27029 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.312721 27472 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.835987 27498 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:49.893683 27400 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:49.945991 27439 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:49.917075 27249 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:49.837095 27323 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.016374 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:49.920117 27536 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.018803 25030 rpcz_store.cc:276] 0426 13:19:47.244695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.719432 (+474737us) service_pool.cc:225] Handling call
0426 13:19:47.719755 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.720117 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.720145 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.720619 (+ 474us) log.cc:842] Serialized 26 byte log entry
0426 13:19:48.391749 (+671130us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:48.391790 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:48.391876 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:49.151483 (+759607us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:49.885557 (+734074us) raft_consensus.cc:1671] finished
0426 13:19:49.885577 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:49.885780 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19321,"raft.queue_time_us":192729,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":14549,"spinlock_wait_cycles":336450432,"thread_start_us":670761,"threads_started":1,"wal-append.queue_time_us":891365}
W20240426 13:19:50.004518 27500 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.930s user 0.014s sys 0.000s
I20240426 13:19:49.897634 26945 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.022733 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:50.021893 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 198389us
I20240426 13:19:50.026484 25026 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
W20240426 13:19:50.018558 27511 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240426 13:19:49.860553 27461 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:19:50.019032 27475 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:50.048887 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.028s sys 0.057s
I20240426 13:19:50.019258 27524 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.021440 27502 consensus_queue.cc:1225] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:19:49.903172 27113 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.052695 25623 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.054078 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.124426 25103 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:50.124538 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1450) took 1113 ms. Trace:
I20240426 13:19:50.125233 25023 rpcz_store.cc:276] 0426 13:19:49.011442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.046290 (+1034848us) service_pool.cc:225] Handling call
0426 13:19:50.124466 (+ 78176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:49.903517 27483 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.063464 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1831) took 3089 ms. Trace:
I20240426 13:19:50.126860 24953 rpcz_store.cc:276] 0426 13:19:46.974402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.976177 (+ 1775us) service_pool.cc:225] Handling call
0426 13:19:46.976495 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.655319 (+678824us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.063160 (+2407841us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54000a69d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.063392 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1559599872}
I20240426 13:19:50.000779 27540 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:50.066494 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50212.6us
I20240426 13:19:50.133322 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.067027 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:50.117960 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 268828us
I20240426 13:19:50.128391 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.022148 27474 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20240426 13:19:50.052590 27504 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.137090 24953 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:50.136965 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:50.138738 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 330) took 1132 ms. Trace:
I20240426 13:19:50.139251 25030 rpcz_store.cc:276] 0426 13:19:49.006404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.066690 (+1060286us) service_pool.cc:225] Handling call
0426 13:19:50.138690 (+ 72000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.140338 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:50.140689 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.147918 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.149495 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.236411 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 332) took 1227 ms. Trace:
I20240426 13:19:50.237005 25030 rpcz_store.cc:276] 0426 13:19:49.008714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.140400 (+1131686us) service_pool.cc:225] Handling call
0426 13:19:50.236323 (+ 95923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.127900 27491 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.134953 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.250615 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.141244 27276 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.145574 27113 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:50.095321 27500 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:50.271644 27543 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.364s user 0.001s sys 0.000s
I20240426 13:19:50.144430 27376 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.272307 27380 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:50.021360 27554 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20240426 13:19:50.263881 27433 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:50.268460 27491 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.103668 27536 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.279006 25023 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:50.284322 27323 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.261685 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:50.272526 27113 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:50.280876 27380 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:50.290894 27380 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.277529 27551 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:50.285833 25616 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:50.292287 27113 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.296911 25030 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:50.297611 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1448) took 1286 ms. Trace:
I20240426 13:19:50.297983 25030 rpcz_store.cc:276] 0426 13:19:49.010887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.261343 (+1250456us) service_pool.cc:225] Handling call
0426 13:19:50.297541 (+ 36198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.283022 27433 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:50.318240 27545 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.302s user 0.001s sys 0.000s
I20240426 13:19:50.303223 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:50.323282 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 329) took 1317 ms. Trace:
W20240426 13:19:50.323545 25100 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1576) took 3359 ms (client timeout 1602 ms). Trace:
I20240426 13:19:50.323734 25030 rpcz_store.cc:276] 0426 13:19:49.006195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.302247 (+1296052us) service_pool.cc:225] Handling call
0426 13:19:50.323208 (+ 20961us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.324288 25100 rpcz_store.cc:269] 0426 13:19:46.964148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.965948 (+ 1800us) service_pool.cc:225] Handling call
0426 13:19:50.323323 (+3357375us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54003b10d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.323458 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3027776256}
I20240426 13:19:50.294597 26973 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.326100 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 540) took 1929 ms (client timeout 2116 ms). Trace:
I20240426 13:19:50.326267 27555 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:50.326539 25624 rpcz_store.cc:269] 0426 13:19:48.396301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.396432 (+ 131us) service_pool.cc:225] Handling call
0426 13:19:50.325927 (+1929495us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b540053d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.326022 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28568832}
I20240426 13:19:50.327831 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.326134 27556 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20240426 13:19:50.321732 27433 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.288741 27491 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.316183 27497 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.321470 27323 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:50.342499 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.362090 27375 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.364447 25618 raft_consensus.cc:2354] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:19:50.365787 27500 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.370179 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1430) took 3179 ms. Trace:
I20240426 13:19:50.370568 25029 rpcz_store.cc:276] 0426 13:19:47.191008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.474724 (+283716us) service_pool.cc:225] Handling call
0426 13:19:47.999358 (+524634us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:48.005393 (+ 6035us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.369989 (+2364596us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54000b0750. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.370122 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1205726080}
I20240426 13:19:50.371834 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.372864 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:50.373492 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1445) took 1389 ms. Trace:
I20240426 13:19:50.373852 25029 rpcz_store.cc:276] 0426 13:19:48.984504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.371569 (+1387065us) service_pool.cc:225] Handling call
0426 13:19:50.373428 (+ 1859us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.374847 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.336846 27439 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.375779 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1447) took 1374 ms. Trace:
I20240426 13:19:50.381063 25029 rpcz_store.cc:276] 0426 13:19:49.001861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.374600 (+1372739us) service_pool.cc:225] Handling call
0426 13:19:50.375718 (+ 1118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.327072 27555 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:50.363876 27249 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.382411 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.385246 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 334) took 1365 ms. Trace:
I20240426 13:19:50.385797 25029 rpcz_store.cc:276] 0426 13:19:49.019732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.382118 (+1362386us) service_pool.cc:225] Handling call
0426 13:19:50.385181 (+ 3063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.358752 27497 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:50.340246 27540 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.388355 27249 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.387858 27552 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20240426 13:19:50.378877 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.396178 27447 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.621s user 0.002s sys 0.008s
I20240426 13:19:50.396656 27447 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.397938 25621 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:50.409684 27483 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.411538 27549 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.396s user 0.000s sys 0.000s
I20240426 13:19:50.411487 27553 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:50.415498 27375 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.418197 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1948) took 1200 ms. Trace:
I20240426 13:19:50.418675 25023 rpcz_store.cc:276] 0426 13:19:49.217458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.126215 (+908757us) service_pool.cc:225] Handling call
0426 13:19:50.417977 (+291762us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54004155d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.418124 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333289728}
I20240426 13:19:50.424674 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.393741 27491 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.425721 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 542) took 1925 ms (client timeout 1754 ms). Trace:
W20240426 13:19:50.434533 25622 rpcz_store.cc:269] 0426 13:19:48.500605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.500803 (+ 198us) service_pool.cc:225] Handling call
0426 13:19:50.424481 (+1923678us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:19:50.425639 (+ 1158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29923712}
I20240426 13:19:50.435787 25023 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:50.436712 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1438) took 1502 ms. Trace:
I20240426 13:19:50.437196 25023 rpcz_store.cc:276] 0426 13:19:48.934158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.419686 (+1485528us) service_pool.cc:225] Handling call
0426 13:19:50.426712 (+ 7026us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.436397 (+ 9685us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.436653 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.333374 27400 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.442950 25023 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:50.325196 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:50.446480 27366 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:50.478374 27366 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:50.446552 27550 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.285s user 0.001s sys 0.000s
I20240426 13:19:50.410760 26839 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.448894 27555 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.449465 27447 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.449509 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1944) took 1233 ms. Trace:
I20240426 13:19:50.509565 25028 rpcz_store.cc:276] 0426 13:19:49.216493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.818422 (+601929us) service_pool.cc:225] Handling call
0426 13:19:50.449319 (+630897us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.449452 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179757440}
I20240426 13:19:50.453027 26945 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.453794 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 398) took 2859 ms. Trace:
I20240426 13:19:50.513690 24952 rpcz_store.cc:276] 0426 13:19:47.594372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.598566 (+ 4194us) service_pool.cc:225] Handling call
0426 13:19:47.598875 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:47.647148 (+ 48273us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.647179 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:47.647230 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:47.647299 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.453553 (+2806254us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.453607 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.453734 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110426880}
I20240426 13:19:50.455552 27391 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.458902 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 424) took 1508 ms. Trace:
I20240426 13:19:50.466346 25030 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:50.466374 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1440) took 1520 ms. Trace:
I20240426 13:19:50.515468 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1454) took 1357 ms. Trace:
I20240426 13:19:50.477819 27400 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.515267 25098 rpcz_store.cc:276] 0426 13:19:48.950194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.996570 (+ 46376us) service_pool.cc:225] Handling call
0426 13:19:48.996863 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.003577 (+ 6714us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:49.049908 (+ 46331us) log.cc:842] Serialized 14 byte log entry
0426 13:19:49.219068 (+169160us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:49.219165 (+ 97us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:49.219208 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.458613 (+1239405us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.458672 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.458833 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39857,"prepare.queue_time_us":51503,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":36686,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7348,"spinlock_wait_cycles":14805376,"thread_start_us":164842,"threads_started":2,"wal-append.queue_time_us":558632}
I20240426 13:19:50.412446 27553 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.479348 27366 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.506666 27360 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:50.431450 27113 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.515467 25023 rpcz_store.cc:276] 0426 13:19:48.946316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.438561 (+1492245us) service_pool.cc:225] Handling call
0426 13:19:50.442697 (+ 4136us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.466058 (+ 23361us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.466310 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.516171 25030 rpcz_store.cc:276] 0426 13:19:49.158611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.324918 (+1166307us) service_pool.cc:225] Handling call
0426 13:19:50.515414 (+190496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.523582 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1441) took 1573 ms. Trace:
W20240426 13:19:50.516952 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.019s sys 0.038s
I20240426 13:19:50.460201 27433 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:50.525645 25028 rpcz_store.cc:276] 0426 13:19:48.950401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.510616 (+1560215us) service_pool.cc:225] Handling call
0426 13:19:50.522933 (+ 12317us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.523166 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.523220 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.523277 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.523320 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.523386 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.523518 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.527766 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s user 0.019s sys 0.038s
I20240426 13:19:50.525871 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 323) took 1527 ms. Trace:
I20240426 13:19:50.528266 25030 rpcz_store.cc:276] 0426 13:19:48.998543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.525160 (+1526617us) service_pool.cc:225] Handling call
0426 13:19:50.525429 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.525579 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.525602 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.525651 (+ 49us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:50.525683 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.525735 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.525827 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.527427 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 325) took 1528 ms. Trace:
I20240426 13:19:50.530772 25028 rpcz_store.cc:276] 0426 13:19:48.999003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.526674 (+1527671us) service_pool.cc:225] Handling call
0426 13:19:50.526990 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.527144 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.527168 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.527211 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:50.527241 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.527289 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.527381 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11648}
I20240426 13:19:50.527268 27366 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:50.533807 27372 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:50.534086 27113 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.453954 27524 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.522414 27391 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:50.551348 25024 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:50.542940 27518 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.570276 25028 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:19:50.575201 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1449) took 1564 ms. Trace:
I20240426 13:19:50.575693 25028 rpcz_store.cc:276] 0426 13:19:49.011191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.531805 (+1520614us) service_pool.cc:225] Handling call
0426 13:19:50.569962 (+ 38157us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.574951 (+ 4989us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.575149 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.530253 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 327) took 1530 ms. Trace:
W20240426 13:19:50.577278 25621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 543) took 1455 ms (client timeout 1802 ms). Trace:
I20240426 13:19:50.579119 25030 rpcz_store.cc:276] 0426 13:19:48.999472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.529406 (+1529934us) service_pool.cc:225] Handling call
0426 13:19:50.529708 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.529892 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.529927 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.529978 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:50.530020 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.530081 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.530198 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.580471 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 423) took 1632 ms. Trace:
I20240426 13:19:50.581570 25029 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:19:50.583665 25103 rpcz_store.cc:276] 0426 13:19:48.947990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.951372 (+ 3382us) service_pool.cc:225] Handling call
0426 13:19:50.580255 (+1628883us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.580408 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2641634560}
I20240426 13:19:50.581766 25028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:50.586066 25028 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
W20240426 13:19:50.582139 24948 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.019s sys 0.090s
I20240426 13:19:50.583002 25023 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240426 13:19:50.586704 25028 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:50.585306 25030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:51366
I20240426 13:19:50.589660 25030 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240426 13:19:50.590096 25030 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.589542 24948 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s user 0.019s sys 0.090s
I20240426 13:19:50.592101 24951 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.592856 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1536) took 3147 ms. Trace:
I20240426 13:19:50.593251 24951 rpcz_store.cc:276] 0426 13:19:47.445017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.488578 (+ 43561us) service_pool.cc:225] Handling call
0426 13:19:50.591639 (+3103061us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:50.591874 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.592584 (+ 710us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.592795 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7502720}
I20240426 13:19:50.594426 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1853) took 1388 ms. Trace:
I20240426 13:19:50.591300 27113 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:50.602798 27113 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.603555 27524 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.604691 27113 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.602813 27507 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:50.606150 27507 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:50.606843 27507 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.608706 27507 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.579942 25621 rpcz_store.cc:269] 0426 13:19:49.122211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.128157 (+ 5946us) service_pool.cc:225] Handling call
0426 13:19:50.550595 (+1422438us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:50.574482 (+ 23887us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2708740864}
I20240426 13:19:50.587884 27433 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:50.615916 27489 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.598909 24948 rpcz_store.cc:276] 0426 13:19:49.206034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.207485 (+ 1451us) service_pool.cc:225] Handling call
0426 13:19:49.207952 (+ 467us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.827203 (+619251us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.827235 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.827754 (+ 519us) log.cc:842] Serialized 52 byte log entry
0426 13:19:50.003203 (+175449us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.592012 (+588809us) log.cc:842] Serialized 14 byte log entry
0426 13:19:50.594050 (+ 2038us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.594138 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:50.594192 (+ 54us) raft_consensus.cc:1671] finished
0426 13:19:50.594210 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.594354 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3725,"prepare.queue_time_us":18889,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":131831,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":57335,"thread_start_us":680332,"threads_started":2,"wal-append.queue_time_us":420419}
I20240426 13:19:50.631072 27391 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.621198 27553 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.648443 27518 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.655200 24920 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.040s sys 0.053s
W20240426 13:19:50.655718 24920 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.041s sys 0.054s
W20240426 13:19:50.656075 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 757730us
W20240426 13:19:50.660133 24954 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.021s sys 0.027s
W20240426 13:19:50.660602 24954 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.021s sys 0.027s
I20240426 13:19:50.672601 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1850) took 1503 ms. Trace:
I20240426 13:19:50.673135 24954 rpcz_store.cc:276] 0426 13:19:49.169448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.170867 (+ 1419us) service_pool.cc:225] Handling call
0426 13:19:49.216615 (+ 45748us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.233081 (+ 16466us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.233122 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.233637 (+ 515us) log.cc:842] Serialized 52 byte log entry
0426 13:19:50.048568 (+814931us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:50.058442 (+ 9874us) log.cc:842] Serialized 14 byte log entry
0426 13:19:50.409566 (+351124us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:50.448165 (+ 38599us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b1800330880. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:19:50.672203 (+224038us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.672281 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:50.672343 (+ 62us) raft_consensus.cc:1671] finished
0426 13:19:50.672362 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.672537 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6726,"prepare.queue_time_us":19343,"prepare.run_cpu_time_us":288,"prepare.run_wall_time_us":5418,"raft.queue_time_us":94232,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":7897,"spinlock_wait_cycles":7952896,"thread_start_us":1348579,"threads_started":2,"wal-append.queue_time_us":651941}
I20240426 13:19:50.618814 27380 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.681494 27380 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:50.671715 27422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.027s sys 0.070s
I20240426 13:19:50.687474 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.671607 27518 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.690016 27518 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.690408 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.686499 27422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.027s sys 0.070s
I20240426 13:19:50.693284 25027 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:50.691879 27518 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:50.694288 27465 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.695098 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1435) took 1779 ms. Trace:
I20240426 13:19:50.695454 25027 rpcz_store.cc:276] 0426 13:19:48.915279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.922202 (+ 6923us) service_pool.cc:225] Handling call
0426 13:19:48.922536 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.693686 (+1771150us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.694878 (+ 1192us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002ad354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:50.695046 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4071686144}
I20240426 13:19:50.697870 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1942) took 1481 ms. Trace:
I20240426 13:19:50.698931 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.698346 27490 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.702296 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.704260 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:50.706056 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:50.694896 27465 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:50.700588 25027 rpcz_store.cc:276] 0426 13:19:49.215969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.697037 (+1481068us) service_pool.cc:225] Handling call
0426 13:19:50.697341 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.697519 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:50.697552 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:50.697613 (+ 61us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:50.697653 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:50.697712 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:50.697815 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:50.705701 25030 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:50.718724 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.004s sys 0.018s
W20240426 13:19:50.723551 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.662s user 0.004s sys 0.018s
W20240426 13:19:50.724015 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 662822us
I20240426 13:19:50.730345 25028 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:50.731587 25027 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:50.706248 25616 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:50.755687 25030 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:50.758700 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.759333 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:50.759600 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.725780 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:50.760491 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:50.761524 24948 raft_consensus.cc:2396] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240426 13:19:50.758010 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:50.708187 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:50.757455 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.006s sys 0.034s
I20240426 13:19:51.014122 24947 raft_consensus.cc:2354] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
W20240426 13:19:51.014735 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.010s sys 0.037s
W20240426 13:19:51.015129 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 992412us
I20240426 13:19:50.787461 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51366 (request call id 1941) took 1546 ms. Trace:
I20240426 13:19:51.016024 25030 rpcz_store.cc:276] 0426 13:19:49.215726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.583700 (+1367974us) service_pool.cc:225] Handling call
0426 13:19:50.762491 (+178791us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:50.762799 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.715s user 0.031s sys 0.014s
W20240426 13:19:51.018327 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.970s user 0.033s sys 0.019s
W20240426 13:19:50.983039 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.017s sys 0.015s
W20240426 13:19:51.020416 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.018s sys 0.016s
W20240426 13:19:51.021075 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 871630us
I20240426 13:19:51.021967 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.825256 25028 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.023545 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.024626 24953 raft_consensus.cc:2379] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:51.025629 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.026851 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.023545 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:50.840837 25615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s user 0.009s sys 0.040s
W20240426 13:19:51.034616 25615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.097s user 0.009s sys 0.056s
I20240426 13:19:50.764607 24952 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20240426 13:19:51.035215 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:50.770659 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.007s sys 0.028s
W20240426 13:19:51.038030 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.897s user 0.019s sys 0.047s
I20240426 13:19:51.038690 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:51366 (request call id 1940) took 1807 ms. Trace:
I20240426 13:19:50.831346 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:51.039534 25028 rpcz_store.cc:276] 0426 13:19:49.215453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.577020 (+1361567us) service_pool.cc:225] Handling call
0426 13:19:51.023287 (+446267us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:19:51.041091 24955 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.096s user 0.010s sys 0.016s
I20240426 13:19:51.041298 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 868) took 1896 ms. Trace:
W20240426 13:19:51.039677 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 899382us
W20240426 13:19:51.032855 27029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.006s sys 0.020s
W20240426 13:19:50.862509 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.001s sys 0.018s
W20240426 13:19:51.043365 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.005s sys 0.021s
I20240426 13:19:51.041671 25030 rpcz_store.cc:276] 0426 13:19:49.144848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.039033 (+1894185us) service_pool.cc:225] Handling call
0426 13:19:51.041257 (+ 2224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:51.045105 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046356 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046784 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.046962 25028 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.047351 25030 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 6
W20240426 13:19:50.862757 25097 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.959s user 0.027s sys 0.042s
I20240426 13:19:51.050101 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.041440 24955 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.096s user 0.010s sys 0.016s
I20240426 13:19:51.048889 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.049113 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:51.038877 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 313146us
I20240426 13:19:50.829102 27508 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:50.841413 27564 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:50.911764 27486 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.174s user 0.008s sys 0.037s
W20240426 13:19:50.862684 27515 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.009s sys 0.010s
W20240426 13:19:51.050733 25097 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.147s user 0.029s sys 0.048s
I20240426 13:19:51.052232 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.055955 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:50.761978 27553 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.067451 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.040954 27197 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.015s sys 0.028s
I20240426 13:19:51.061890 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:51.065797 27486 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.331s user 0.015s sys 0.038s
I20240426 13:19:50.788638 27360 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:50.788185 27420 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:50.755434 27372 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.296s
W20240426 13:19:51.043233 27029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s user 0.006s sys 0.020s
I20240426 13:19:51.058821 25096 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:51.094818 25103 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.074616 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1594) took 1186 ms. Trace:
I20240426 13:19:50.719275 27490 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.096536 27564 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.050964 27479 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:51.094749 25105 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.116650 27564 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.117959 25104 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1430) took 2164 ms (client timeout 2180 ms). Trace:
I20240426 13:19:51.118325 27271 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:51.118880 27271 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:51.057550 27429 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.120122 27429 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.073880 27515 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.023s user 0.011s sys 0.015s
I20240426 13:19:51.068495 27366 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:51.098758 27197 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.075s user 0.016s sys 0.028s
I20240426 13:19:51.062822 27524 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.098356 27553 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.113252 27565 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20240426 13:19:51.063246 27435 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.124426 25620 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.078431 26824 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20240426 13:19:51.106897 27360 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:51.119853 27271 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.066982 27508 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.151098 27271 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.130911 27366 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.154474 27366 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.132565 27524 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.155928 27524 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.161401 27508 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.161965 27503 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.428s user 0.009s sys 0.016s
I20240426 13:19:51.162392 27553 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.163520 27490 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.164180 27429 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.164533 27465 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.165594 27470 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:51.166126 27470 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
W20240426 13:19:51.166658 27507 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.557s user 0.008s sys 0.000s
I20240426 13:19:51.167119 27470 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.170640 27496 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.178622 27498 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240426 13:19:51.179018 27498 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:51.179414 27494 log.cc:924] Time spent T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.527s user 0.007s sys 0.017s
I20240426 13:19:51.179801 27498 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.181061 27498 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 1 FOLLOWER]: Advancing to term 2
I20240426 13:19:50.790335 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.225684 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:51.226495 25024 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.016s sys 0.006s
W20240426 13:19:51.227151 25024 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.615s user 0.016s sys 0.006s
W20240426 13:19:51.229324 24995 connection.cc:232] RPC call timeout handler was delayed by -1.5798693399999593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20240426 13:19:51.051586 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1855) took 1819 ms. Trace:
W20240426 13:19:51.229985 24995 connection.cc:232] RPC call timeout handler was delayed by -1.575369339999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20240426 13:19:51.230477 24995 connection.cc:232] RPC call timeout handler was delayed by -1.5627693400000453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20240426 13:19:51.066381 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.096308 25098 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.125628 25105 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:51.118360 25104 rpcz_store.cc:269] 0426 13:19:48.953832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.964411 (+ 10579us) service_pool.cc:225] Handling call
0426 13:19:51.117772 (+2153361us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.117900 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2117535488}
W20240426 13:19:51.231215 24995 connection.cc:232] RPC call timeout handler was delayed by -1.2135693399999354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
I20240426 13:19:51.245966 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.230528 24955 rpcz_store.cc:276] 0426 13:19:49.232295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.232481 (+ 186us) service_pool.cc:225] Handling call
0426 13:19:49.234580 (+ 2099us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:49.234932 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:49.234970 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:49.244199 (+ 9229us) log.cc:842] Serialized 52 byte log entry
0426 13:19:49.886749 (+642550us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:49.917607 (+ 30858us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.051253 (+1133646us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.051323 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:51.051366 (+ 43us) raft_consensus.cc:1671] finished
0426 13:19:51.051378 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.051531 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38445,"prepare.queue_time_us":31305,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":212056,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":8942,"spinlock_wait_cycles":715136,"thread_start_us":1747593,"threads_started":2,"wal-append.queue_time_us":647644}
I20240426 13:19:51.242650 25097 rpcz_store.cc:276] 0426 13:19:49.887692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.887984 (+ 292us) service_pool.cc:225] Handling call
0426 13:19:49.888322 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.888592 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:19:49.903393 (+ 14801us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.073744 (+1170351us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.073896 (+ 152us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:51.074023 (+ 127us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.074181 (+ 158us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.074476 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1268788,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":310,"raft.queue_time_us":42768,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":6928,"thread_start_us":1169747,"threads_started":1}
W20240426 13:19:51.242451 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.000s sys 0.037s
W20240426 13:19:51.251200 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.162s user 0.000s sys 0.038s
I20240426 13:19:51.225687 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:51.254803 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 188353us
I20240426 13:19:51.255074 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1851) took 2074 ms. Trace:
I20240426 13:19:51.255640 24956 rpcz_store.cc:276] 0426 13:19:49.180169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.181754 (+ 1585us) service_pool.cc:225] Handling call
0426 13:19:51.254698 (+2072944us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54003c32d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.254872 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1405855488}
W20240426 13:19:51.242169 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 186106us
I20240426 13:19:51.241262 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1922) took 3959 ms. Trace:
I20240426 13:19:51.258086 25031 rpcz_store.cc:276] 0426 13:19:47.281368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.792351 (+510983us) service_pool.cc:225] Handling call
0426 13:19:47.849365 (+ 57014us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:47.857063 (+ 7698us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:47.857105 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:47.857608 (+ 503us) log.cc:842] Serialized 26 byte log entry
0426 13:19:47.973300 (+115692us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:47.973342 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:47.973410 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:51.178884 (+3205474us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:51.240315 (+ 61431us) raft_consensus.cc:1671] finished
0426 13:19:51.240387 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.240935 (+ 548us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7307,"raft.queue_time_us":157732,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":365789440,"thread_start_us":80053,"threads_started":1,"wal-append.queue_time_us":730549}
I20240426 13:19:51.259668 26824 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.259860 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 403) took 2638 ms. Trace:
W20240426 13:19:51.260278 26679 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.012s sys 0.032s
I20240426 13:19:51.266337 24950 rpcz_store.cc:276] 0426 13:19:48.621704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.630346 (+ 8642us) service_pool.cc:225] Handling call
0426 13:19:48.653552 (+ 23206us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:48.742455 (+ 88903us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:48.742496 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:48.742545 (+ 49us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:48.742577 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.259411 (+2516834us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.259484 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.259701 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165910272}
I20240426 13:19:51.285998 27360 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:51.286226 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.287230 25031 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 7
W20240426 13:19:51.288272 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 64296.5us
W20240426 13:19:51.288475 26679 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.588s user 0.014s sys 0.032s
I20240426 13:19:51.294250 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 580) took 1406 ms. Trace:
I20240426 13:19:51.294842 25620 rpcz_store.cc:276] 0426 13:19:49.887433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.887551 (+ 118us) service_pool.cc:225] Handling call
0426 13:19:49.887831 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:51.132689 (+1244858us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.293560 (+160871us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540065e854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.293953 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2843583744}
I20240426 13:19:51.295331 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.297220 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 550) took 2062 ms. Trace:
I20240426 13:19:51.297713 25623 rpcz_store.cc:276] 0426 13:19:49.234562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.816690 (+582128us) service_pool.cc:225] Handling call
0426 13:19:49.817061 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.134122 (+317061us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.296653 (+1162531us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400436554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.296983 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541475072}
W20240426 13:19:51.297600 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52008.8us
I20240426 13:19:51.227926 24921 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7.
I20240426 13:19:51.305055 27481 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240426 13:19:51.305267 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 77827.8us
I20240426 13:19:51.305742 27481 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.306386 27481 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:51.306442 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:51.299779 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.287134 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1425) took 2343 ms. Trace:
I20240426 13:19:51.312633 25099 rpcz_store.cc:276] 0426 13:19:48.944014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.944163 (+ 149us) service_pool.cc:225] Handling call
0426 13:19:48.944426 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.812742 (+868316us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.262728 (+1449986us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54001b3354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.286890 (+ 24162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1912283520}
W20240426 13:19:51.316529 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58378.8us
I20240426 13:19:51.325805 27492 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:51.331168 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.339193 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.343312 25098 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:51.338272 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.343840 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:51.356184 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.333825 27492 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.360095 27505 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.361178 27496 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20240426 13:19:51.361085 27505 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.362929 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1947) took 2145 ms. Trace:
I20240426 13:19:51.363512 25027 rpcz_store.cc:276] 0426 13:19:49.217221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.709063 (+1491842us) service_pool.cc:225] Handling call
0426 13:19:50.709375 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.731935 (+ 22560us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.362223 (+630288us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b54002fa654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.362691 (+ 468us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50729856}
I20240426 13:19:51.368729 27475 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.368955 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.367916 25105 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.370138 24921 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.372061 27492 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.375929 27498 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:51.377074 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46156.9us
I20240426 13:19:51.379936 25099 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.379206 27498 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240426 13:19:51.383913 27505 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.384156 27507 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:51.384802 27507 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.385207 27507 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.388340 26713 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:51.388823 26713 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.389245 26713 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:51.390218 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1902) took 5383 ms (client timeout 1504 ms). Trace:
W20240426 13:19:51.390717 25025 rpcz_store.cc:269] 0426 13:19:46.006668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:46.006811 (+ 143us) service_pool.cc:225] Handling call
0426 13:19:51.389761 (+5382950us) spinlock_profiling.cc:243] Waited 2.76 s on lock 0x7b540025f154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.389978 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6341508864}
I20240426 13:19:51.377419 26969 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.393343 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1444) took 2417 ms. Trace:
I20240426 13:19:51.394035 25023 rpcz_store.cc:276] 0426 13:19:48.975678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.525124 (+1549446us) service_pool.cc:225] Handling call
0426 13:19:50.526951 (+ 1827us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.588279 (+ 61328us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.392756 (+804477us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.393138 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126969344}
I20240426 13:19:51.394724 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1437) took 2465 ms. Trace:
I20240426 13:19:51.395157 25029 rpcz_store.cc:276] 0426 13:19:48.929675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:50.390892 (+1461217us) service_pool.cc:225] Handling call
0426 13:19:50.391303 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.585667 (+194364us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.394310 (+808643us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400096fd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.394542 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437067648}
I20240426 13:19:51.396327 25097 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:51.357331 27474 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.400527 27093 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.401561 27093 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.399638 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1920) took 4155 ms. Trace:
I20240426 13:19:51.401028 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.413097 25103 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:51.413838 27474 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.415823 27350 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:51.416258 27350 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:51.417382 27350 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.420563 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 80304.4us
I20240426 13:19:51.421952 25102 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.422360 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:51.425251 27310 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:51.426769 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47496.1us
I20240426 13:19:51.427204 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1426) took 2482 ms. Trace:
I20240426 13:19:51.427623 25101 rpcz_store.cc:276] 0426 13:19:48.944353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.944494 (+ 141us) service_pool.cc:225] Handling call
0426 13:19:51.426762 (+2482268us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54003b06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.427012 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832645888}
I20240426 13:19:51.432457 25025 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:51.435084 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 92332.5us
I20240426 13:19:51.436609 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 541) took 2995 ms. Trace:
I20240426 13:19:51.437057 25617 rpcz_store.cc:276] 0426 13:19:48.441063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:48.460291 (+ 19228us) service_pool.cc:225] Handling call
0426 13:19:48.546765 (+ 86474us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:49.021202 (+474437us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.436005 (+2414803us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54006d0bd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:51.436409 (+ 404us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":902578048}
W20240426 13:19:51.437788 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 374) took 4296 ms (client timeout 1990 ms). Trace:
W20240426 13:19:51.438192 25619 rpcz_store.cc:269] 0426 13:19:47.140951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.141078 (+ 127us) service_pool.cc:225] Handling call
0426 13:19:51.437432 (+4296354us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54002a58d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.437598 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4260871936}
I20240426 13:19:51.443681 27496 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.444397 27496 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.369701 27475 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.446630 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1442) took 2480 ms (client timeout 2817 ms). Trace:
W20240426 13:19:51.447221 25026 rpcz_store.cc:269] 0426 13:19:48.966209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.835219 (+869010us) service_pool.cc:225] Handling call
0426 13:19:51.446161 (+1610942us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54001e4954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.446384 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311917440}
I20240426 13:19:51.465354 25031 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:51.465780 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 166194us
I20240426 13:19:51.469674 27490 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.465735 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.469863 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.474372 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:51.476259 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54091.9us
I20240426 13:19:51.404561 25024 rpcz_store.cc:276] 0426 13:19:47.244312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.630933 (+386621us) service_pool.cc:225] Handling call
0426 13:19:47.639948 (+ 9015us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:50.550949 (+2911001us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b58004e05b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:19:50.607790 (+ 56841us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:51.228006 (+620216us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.365133 (+137127us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.370133 (+ 5000us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:51.377932 (+ 7799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.398974 (+ 21042us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.399577 (+ 603us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8301,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":14415,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"spinlock_wait_cycles":374286080,"thread_start_us":733546,"threads_started":2}
I20240426 13:19:51.480513 27499 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.479811 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:51.481619 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.483412 27482 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:51.483628 27490 consensus_queue.cc:1046] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:51.483738 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.487929 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:51.485199 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 331) took 2478 ms (client timeout 1525 ms). Trace:
I20240426 13:19:51.484390 26969 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:51.507776 25022 rpcz_store.cc:269] 0426 13:19:49.006577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.014708 (+ 8131us) service_pool.cc:225] Handling call
0426 13:19:51.484756 (+2470048us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:51.484947 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498607360}
I20240426 13:19:51.515491 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.516875 25098 raft_consensus.cc:2379] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:51.520186 26679 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:51.520807 26679 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:51.521209 26679 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.506078 27481 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.505234 27093 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:51.525490 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57284.8us
I20240426 13:19:51.527004 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.527021 27492 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:51.528136 27482 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.529500 27481 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.532837 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49268.7us
I20240426 13:19:51.535897 27386 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.534303 27492 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:51.535437 27482 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.487269 27474 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.538296 27107 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.537765 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:51.540274 25070 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:51.535359 25616 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.707s user 0.011s sys 0.007s
I20240426 13:19:51.544652 27271 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:51.545082 27492 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:51.546053 27470 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.556072 25096 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.556911 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:51.558933 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.559619 27379 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:51.560166 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:51.542263 27350 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:51.487591 27495 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.561857 25096 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.552619 27107 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.571952 25025 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240426 13:19:51.602838 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.600489 25098 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:51.606683 25098 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:51.562733 27379 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.610245 25619 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:51.611006 25028 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.502s user 0.012s sys 0.003s
I20240426 13:19:51.611402 25028 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:51.612406 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6231911789999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
I20240426 13:19:51.616669 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:51.611713 27107 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.610270 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 547) took 2426 ms. Trace:
I20240426 13:19:51.626327 24853 catalog_manager.cc:5484] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.578014 27475 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.630806 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 72090.3us
I20240426 13:19:51.628190 27416 raft_consensus.cc:3032] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.632234 27416 raft_consensus.cc:729] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.633016 27496 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:51.633193 27416 consensus_queue.cc:260] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.634727 27416 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.635143 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.635918 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.636085 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:51.636400 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.636808 24953 raft_consensus.cc:2379] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:51.637827 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.638533 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.636401 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.640522 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:51.640792 24950 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.641976 25096 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.616761 27327 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:51.643597 27327 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.645407 27426 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:51.645889 27426 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.646728 27426 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.618065 27379 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.647797 27327 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:51.648785 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 60734.8us
I20240426 13:19:51.643105 24995 leader_election.cc:398] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:19:51.650501 24995 leader_election.cc:401] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:51.577327 25616 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:51.640302 25098 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.630818 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93364.4us
I20240426 13:19:51.596638 27150 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:51.661139 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 179903us
I20240426 13:19:51.661309 25102 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:51.562366 26713 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.675159 25102 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1412) took 4479 ms (client timeout 1555 ms). Trace:
W20240426 13:19:51.675668 25102 rpcz_store.cc:269] 0426 13:19:47.187953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:47.188109 (+ 156us) service_pool.cc:225] Handling call
0426 13:19:49.253042 (+2064933us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b5400450720. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902adae1 00007fda9030e4b1 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2
0426 13:19:51.420873 (+2167831us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b580068d8b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda9031b3b8 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c
0426 13:19:51.667541 (+246668us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1260481664}
I20240426 13:19:51.674204 27513 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.680002 24953 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:19:51.631966 27510 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:51.659014 27150 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:51.683243 25098 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.687242 27310 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.687496 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.667286 27386 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:51.619599 25615 rpcz_store.cc:276] 0426 13:19:49.183353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:49.183534 (+ 181us) service_pool.cc:225] Handling call
0426 13:19:49.219519 (+ 35985us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:49.219730 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:19:49.222905 (+ 3175us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.375108 (+2152203us) log.cc:842] Serialized 14 byte log entry
0426 13:19:51.562859 (+187751us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:51.575151 (+ 12292us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:51.578600 (+ 3449us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.599685 (+ 21085us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:51.610099 (+ 10414us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":464,"prepare.queue_time_us":133861,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":73,"raft.queue_time_us":72269,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":9682,"thread_start_us":2188912,"threads_started":4,"wal-append.queue_time_us":737147,"wal-append.run_cpu_time_us":1164,"wal-append.run_wall_time_us":1005237}
W20240426 13:19:51.695436 27461 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.016s sys 0.019s
I20240426 13:19:51.694355 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.681650 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 5, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 5 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:51.698168 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:51.699489 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.689849 27150 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:51.711150 25069 connection.cc:232] RPC call timeout handler was delayed by -1.2973911789999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20240426 13:19:51.711949 25096 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.713012 27526 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:51.713171 27434 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.713678 27526 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.713714 27434 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.712193 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50235.3us
I20240426 13:19:51.689164 27472 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.717394 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.706571 24955 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:51.722365 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:51.706866 27461 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.017s sys 0.020s
I20240426 13:19:51.712424 27498 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:19:51.732241 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62268.3us
I20240426 13:19:51.716009 27526 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.753595 26713 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.754287 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:51.755059 24948 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:51.756605 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.757447 26679 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:51.758582 24854 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39836 (request call id 272) took 18240 ms (client timeout 15000 ms). Trace:
I20240426 13:19:51.759483 26679 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:51.760421 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.761399 24949 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:51.762358 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.763043 25102 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.763615 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.764528 27434 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.765127 25101 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:51.764535 25099 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:51.765830 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:51.766968 27386 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:51.767750 27386 consensus_queue.cc:1225] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:51.768326 27386 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:51.768965 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:51.767587 24996 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:51.679023 27513 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.775137 27475 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:51.775595 27475 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.778407 25096 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:51.768647 27386 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:51.800454 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 88458.2us
I20240426 13:19:51.804756 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:51.777045 24953 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:51.809890 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 201555us
I20240426 13:19:51.814738 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:51.795241 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.734283 27508 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
W20240426 13:19:51.828055 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49599.9us
W20240426 13:19:51.826349 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58915.7us
W20240426 13:19:51.833667 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109484us
I20240426 13:19:51.811893 27472 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:51.832623 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.813534 27386 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:51.842926 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.849830 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:51.777660 27498 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:51.849984 27451 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20240426 13:19:51.855322 27350 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:51.855755 27472 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.862072 27416 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:19:51.874047 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135322us
W20240426 13:19:51.906129 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94000us
W20240426 13:19:51.758962 24854 rpcz_store.cc:269] 0426 13:19:33.518193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:33.527156 (+ 8963us) service_pool.cc:225] Handling call
0426 13:19:51.758346 (+18231190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:46.050869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:46.051519 (+ 650us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:46.051574 (+ 55us) write_op.cc:427] Acquired schema lock
0426 13:19:46.051594 (+ 20us) tablet.cc:660] Decoding operations
0426 13:19:46.052107 (+ 513us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:46.052186 (+ 79us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:46.052215 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0426 13:19:46.052682 (+ 467us) tablet.cc:699] Row locks acquired
0426 13:19:46.052716 (+ 34us) write_op.cc:260] PREPARE: finished
0426 13:19:46.052879 (+ 163us) write_op.cc:270] Start()
0426 13:19:46.053022 (+ 143us) write_op.cc:275] Timestamp: P: 1714137586052815 usec, L: 0
0426 13:19:46.053052 (+ 30us) op_driver.cc:331] REPLICATION: starting
0426 13:19:46.156550 (+103498us) log.cc:842] Serialized 917 byte log entry
0426 13:19:47.040220 (+883670us) op_driver.cc:444] REPLICATION: finished
0426 13:19:47.671830 (+631610us) write_op.cc:300] APPLY: starting
0426 13:19:47.723944 (+ 52114us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:47.775872 (+ 51928us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:47.778046 (+ 2174us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:47.778479 (+ 433us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:47.778671 (+ 192us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0426 13:19:47.778739 (+ 68us) write_op.cc:311] APPLY: finished
0426 13:19:47.779022 (+ 283us) log.cc:842] Serialized 29 byte log entry
0426 13:19:47.779255 (+ 233us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:47.779353 (+ 98us) write_op.cc:446] Released schema lock
0426 13:19:47.779411 (+ 58us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:47.779741 (+ 330us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:47.779783 (+ 42us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4200839,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":564169,"apply.run_cpu_time_us":11354,"apply.run_wall_time_us":130817,"num_ops":2,"prepare.queue_time_us":281434,"prepare.run_cpu_time_us":18948,"prepare.run_wall_time_us":233205,"raft.queue_time_us":353033,"raft.run_cpu_time_us":29712,"raft.run_wall_time_us":497397,"replication_time_us":966628,"thread_start_us":814797,"threads_started":4,"wal-append.queue_time_us":227854,"wal-append.run_cpu_time_us":27520,"wal-append.run_wall_time_us":2374020}]]}
I20240426 13:19:51.871750 27350 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.874910 27508 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:51.940917 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.944634 27410 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:51.945289 27410 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.946326 27410 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.867504 27475 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:51.948311 27487 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20240426 13:19:51.948877 27475 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.950078 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.951062 26996 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.951506 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:51.932150 27470 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.954330 27470 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:51.956465 27470 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:51.957087 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 132791us
I20240426 13:19:51.957521 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.960325 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.960940 25619 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:51.961061 27498 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:51.961586 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:51.962116 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.962904 25024 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:51.899343 26679 raft_consensus.cc:2741] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20240426 13:19:51.963104 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 140117us
I20240426 13:19:51.930912 24853 catalog_manager.cc:5484] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 3 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240426 13:19:51.909742 24920 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:51.961108 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.969601 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:51.932472 27350 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:51.974275 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.973300 25098 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:51.902781 27416 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:51.988909 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:51.989974 25615 raft_consensus.cc:2379] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:51.985273 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:51.973613 27508 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:51.982074 24853 catalog_manager.cc:5484] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:51.997949 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:51.982806 26969 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.982960 26824 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:51.980077 27474 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:51.987147 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 126216us
W20240426 13:19:51.994338 27507 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.609s user 0.014s sys 0.006s
I20240426 13:19:51.988312 27331 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.980799 27310 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.010998 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.011833 27507 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.013801 27507 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.014240 24947 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.015619 27327 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:19:52.016142 27486 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:51.971905 27492 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.017282 27327 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:52.018666 27327 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:52.016245 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:52.021368 27393 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:52.021742 27393 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.022058 27393 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:52.024134 27499 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:52.024703 27499 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:52.025658 27499 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.028594 27472 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.029526 27492 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.039280 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.039990 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:52.006919 26969 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.040951 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.041357 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.041738 26969 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:52.042229 24949 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:52.042945 24953 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:19:52.042033 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.044430 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.037837 24996 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:52.045507 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:52.049785 27438 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.050078 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.050320 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.047024 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:52.050935 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.051384 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.051481 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.052311 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:52.052585 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.053638 24956 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
I20240426 13:19:52.053659 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.054849 27474 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.049785 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.055960 24951 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:52.054407 27513 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.050221 27438 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.062657 27438 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:52.063084 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 93642.6us
I20240426 13:19:52.063892 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.063287 27508 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.066817 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.067432 25025 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:52.067811 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 56915.4us
I20240426 13:19:52.068368 27513 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.051578 24853 catalog_manager.cc:5484] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 4 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:19:52.068802 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.071219 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.051815 25618 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:52.057610 25028 raft_consensus.cc:2362] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5. Re-sending same reply.
I20240426 13:19:52.075181 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.075781 25028 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.076398 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.080675 27474 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.082926 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:19:52.058019 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106644us
I20240426 13:19:51.982687 27271 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20240426 13:19:52.060048 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51643.1us
I20240426 13:19:52.085238 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.056818 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:52.085323 25028 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:51.997459 27496 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20240426 13:19:52.082538 27438 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.086570 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.087746 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.088286 27438 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.088764 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.089380 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.090876 25619 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:52.019182 27327 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:51.998404 27416 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.097501 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:52.103960 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69591.8us
I20240426 13:19:52.097066 25025 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
W20240426 13:19:52.112946 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44289.1us
I20240426 13:19:52.113893 25030 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:52.116520 24949 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:19:52.043499 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:52.129238 27331 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:52.129796 27496 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.130264 27496 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:52.130681 27496 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.059772 26824 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.133280 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.136735 27426 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:52.140161 24953 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:52.117861 27507 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20240426 13:19:52.142128 27507 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:52.141856 27350 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.142549 27507 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:52.144266 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.129829 27331 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.145038 27331 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.145227 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.145505 27350 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.147439 24948 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:52.148630 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.149878 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.150624 24948 raft_consensus.cc:2379] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:52.150777 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:52.151445 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.151767 27376 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.152216 27376 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.152608 27376 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.093271 27107 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.034999 27495 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20240426 13:19:52.123729 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.152021 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:52.166584 25028 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:52.155737 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.169575 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.175163 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.175985 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:52.143429 27507 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.180277 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:52.184218 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 122588us
I20240426 13:19:52.185169 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:52.186965 25030 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 13:19:52.187482 25030 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 2
I20240426 13:19:52.133909 26824 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.164655 27475 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.168704 27107 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:52.192047 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 106931us
I20240426 13:19:52.198731 25030 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 13:19:52.208158 27474 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:52.208726 27474 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.209120 27474 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:52.199213 27327 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.199179 25030 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:52.183838 27434 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.214287 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:52.219504 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.187923 27490 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:52.221470 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 77290.8us
I20240426 13:19:52.216995 26969 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.236224 27434 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.201608 27107 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.218294 25028 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:19:52.254478 25028 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:19:52.254885 25028 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:52.257890 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:52.261415 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60940us
I20240426 13:19:52.262818 24947 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.253947 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:52.289363 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 88470.1us
I20240426 13:19:52.241171 26969 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.253100 27434 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:52.267323 27150 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s user 0.011s sys 0.000s
W20240426 13:19:52.300833 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83019.7us
I20240426 13:19:52.202410 27393 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.327462 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:52.302489 26969 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.335253 24996 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:52.293210 27482 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:52.319355 27487 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:52.354722 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 102512us
I20240426 13:19:52.305959 27093 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20240426 13:19:52.359617 27093 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:52.360059 27093 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:52.360394 27591 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20240426 13:19:52.361338 27501 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:52.361217 27093 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.362006 27303 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.362461 27303 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.362901 27303 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:52.358613 27600 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20240426 13:19:52.365326 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.367164 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.368182 27426 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:52.368816 27426 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:52.369274 27426 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:52.370124 27426 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.268507 27490 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.305399 27507 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.340384 27510 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:52.385310 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:52.386205 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.339933 27327 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.399807 27486 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240426 13:19:52.401091 27486 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:52.401543 27486 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:52.402505 27486 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.311182 27150 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.405827 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.406754 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.411715 25030 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:52.355635 27451 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:52.416076 25102 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:52.419421 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:52.428555 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.429854 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.430987 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.431901 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.371753 27376 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.448045 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.448876 25030 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.455282 25100 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.444867 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:52.432497 27451 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.422526 27498 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:52.391804 27492 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20240426 13:19:52.434517 26679 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:52.375499 27386 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.416734 27513 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20240426 13:19:52.475785 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.476675 25097 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:52.455575 27093 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:52.465250 27451 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:52.467378 26679 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:52.464699 27498 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:52.509675 25030 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:52.494767 27379 consensus_meta.cc:287] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.822s user 0.013s sys 0.000s
I20240426 13:19:52.512495 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:52.520740 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.035s sys 0.041s
W20240426 13:19:52.521171 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.035s sys 0.041s
I20240426 13:19:52.523311 25101 raft_consensus.cc:1226] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.521009 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.526484 25098 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:52.525336 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.538329 25097 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.508267 26679 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:52.508473 27498 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.542407 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:52.549080 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:52.572077 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144581us
I20240426 13:19:52.532368 27490 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.530330 27393 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.582429 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:52.583148 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:52.584237 24947 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:52.584380 27331 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:52.585460 24947 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:19:52.585780 26969 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.589424 27487 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:52.590549 25030 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:52.601541 27376 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:52.605170 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:52.607995 27416 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.586349 27331 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.536120 27461 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:52.615399 25101 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:52.557533 27475 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:52.633026 25069 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:52.638337 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.524883 27379 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.628834 27416 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:52.643314 27508 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.013s sys 0.021s
I20240426 13:19:52.641322 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:52.649137 25096 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.650228 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:52.646914 24952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.027s sys 0.038s
W20240426 13:19:52.686959 24952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.555s user 0.027s sys 0.038s
W20240426 13:19:52.644740 27499 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.016s sys 0.014s
I20240426 13:19:52.643136 27376 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:52.661954 27508 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s user 0.013s sys 0.021s
W20240426 13:19:52.645164 27410 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.014s sys 0.017s
I20240426 13:19:52.648185 27475 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:52.648559 27150 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:52.691052 27499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.017s sys 0.014s
W20240426 13:19:52.702834 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.010s sys 0.027s
I20240426 13:19:52.686724 26969 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.692426 27376 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:52.737673 24853 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.011s sys 0.032s
W20240426 13:19:52.740121 24853 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s user 0.012s sys 0.032s
W20240426 13:19:52.730875 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s user 0.010s sys 0.027s
W20240426 13:19:52.700711 27410 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.015s sys 0.017s
I20240426 13:19:52.656440 27474 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:52.746721 27605 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.279s user 0.000s sys 0.001s
I20240426 13:19:52.647373 27461 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.732738 27386 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.742138 27376 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:52.748214 27606 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
I20240426 13:19:52.738270 27150 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20240426 13:19:52.812800 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 476265us
W20240426 13:19:52.829582 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 451108us
I20240426 13:19:52.868192 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1615) took 1094 ms. Trace:
I20240426 13:19:52.824646 27482 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:52.868821 25100 rpcz_store.cc:276] 0426 13:19:51.759675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.760119 (+ 444us) service_pool.cc:225] Handling call
0426 13:19:52.853761 (+1093642us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b540012d9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:52.853935 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1575075200}
I20240426 13:19:52.878497 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.879191 24947 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:52.880494 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1871) took 1182 ms. Trace:
I20240426 13:19:52.811177 27410 consensus_queue.cc:1046] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20240426 13:19:52.880901 24950 rpcz_store.cc:276] 0426 13:19:51.690616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.690738 (+ 122us) service_pool.cc:225] Handling call
0426 13:19:51.699262 (+ 8524us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:51.709668 (+ 10406us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:51.709709 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:51.710290 (+ 581us) log.cc:842] Serialized 26 byte log entry
0426 13:19:51.994590 (+284300us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:51.994622 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:51.994682 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.873018 (+878336us) raft_consensus.cc:1671] finished
0426 13:19:52.873043 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.873220 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2667,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":283931,"threads_started":1,"wal-append.queue_time_us":744658}
I20240426 13:19:52.838577 27510 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:52.886476 24948 raft_consensus.cc:1226] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:52.886457 27507 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.815824 27303 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.855559 27499 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.820792 26996 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
W20240426 13:19:52.882951 27418 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.020s sys 0.023s
I20240426 13:19:52.903542 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.878944 27379 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:52.922309 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 400530us
I20240426 13:19:52.906064 27410 consensus_queue.cc:1225] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:19:52.919725 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:52.928247 24950 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.933483 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:52.922678 27418 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.020s sys 0.024s
I20240426 13:19:52.924690 27416 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.946413 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:52.926517 27438 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:52.933821 27350 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:52.953457 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.955252 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.957304 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.958812 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:52.959621 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1953) took 1002 ms. Trace:
I20240426 13:19:52.959982 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.965281 27451 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:52.966928 25099 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:52.967012 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1956) took 1008 ms. Trace:
I20240426 13:19:52.967839 27451 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:52.960002 25029 rpcz_store.cc:276] 0426 13:19:51.957446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.960174 (+ 2728us) service_pool.cc:225] Handling call
0426 13:19:51.960471 (+ 297us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:51.987839 (+ 27368us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:51.987879 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:51.988380 (+ 501us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.521671 (+533291us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:52.912218 (+390547us) log.cc:842] Serialized 14 byte log entry
0426 13:19:52.923773 (+ 11555us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.924268 (+ 495us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.959392 (+ 35124us) raft_consensus.cc:1671] finished
0426 13:19:52.959422 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.959566 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24969,"prepare.queue_time_us":356016,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":34361,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":11688,"thread_start_us":891332,"threads_started":2,"wal-append.queue_time_us":778420}
I20240426 13:19:52.978281 25029 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:52.967552 25027 rpcz_store.cc:276] 0426 13:19:51.958442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.958570 (+ 128us) service_pool.cc:225] Handling call
0426 13:19:51.958918 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.005190 (+ 46272us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.005230 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.060983 (+ 55753us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.220737 (+159754us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b180043bca0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:52.573673 (+352936us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.573724 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.573803 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:52.966665 (+392862us) raft_consensus.cc:1671] finished
0426 13:19:52.966693 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:52.966944 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37302,"raft.queue_time_us":44853,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":51718,"spinlock_wait_cycles":68600704,"thread_start_us":352757,"threads_started":1,"wal-append.queue_time_us":792752}
I20240426 13:19:52.996204 25027 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:52.936434 27501 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:53.031450 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 393364us
W20240426 13:19:53.032374 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 621852us
I20240426 13:19:52.923506 27499 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.037150 24921 connection.cc:232] RPC call timeout handler was delayed by -1.7364382590000105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20240426 13:19:53.037647 24921 connection.cc:232] RPC call timeout handler was delayed by -1.5960382590000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17110000000000003s for a response
I20240426 13:19:52.930217 27507 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.037935 24921 connection.cc:232] RPC call timeout handler was delayed by -1.4142382589999443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20240426 13:19:53.038870 24921 connection.cc:232] RPC call timeout handler was delayed by -1.3892003540000815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
W20240426 13:19:53.039335 24921 connection.cc:232] RPC call timeout handler was delayed by -1.2621382590000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20240426 13:19:53.039530 27507 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.938372 27379 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20240426 13:19:52.955408 27410 raft_consensus.cc:3032] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:52.952576 27474 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:52.953058 27416 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.035241 27501 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.039659 27303 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.047019 27638 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.043709 27107 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.743s user 0.007s sys 0.002s
I20240426 13:19:53.026916 27350 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.060125 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1955) took 1087 ms. Trace:
I20240426 13:19:53.042802 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.075387 25031 rpcz_store.cc:276] 0426 13:19:51.958114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:51.961176 (+ 3062us) service_pool.cc:225] Handling call
0426 13:19:51.961457 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.008184 (+ 46727us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.008216 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.046699 (+ 38483us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.749650 (+702951us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.749702 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.749797 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.044897 (+295100us) raft_consensus.cc:1671] finished
0426 13:19:53.044925 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.045070 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53574,"raft.queue_time_us":44744,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":19837,"thread_start_us":571571,"threads_started":1,"wal-append.queue_time_us":807718}
I20240426 13:19:53.072451 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.076294 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.082507 25027 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.063963 24950 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:19:53.092584 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57120.1us
I20240426 13:19:52.951458 27510 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.095504 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.096019 27350 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.096146 25025 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:53.073863 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.097570 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:52.946543 27482 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.098662 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.099414 25026 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.072674 26679 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.046365 27461 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:53.102491 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 60155.9us
I20240426 13:19:53.102746 27526 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:19:53.103281 27526 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.103652 27647 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.104542 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.105059 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.105845 25030 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.106657 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.107654 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.068758 27416 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.109238 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.110458 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:53.057847 27434 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.731s user 0.000s sys 0.008s
I20240426 13:19:53.112282 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.112421 27434 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.113061 25029 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20240426 13:19:53.113663 27496 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.983s user 0.010s sys 0.001s
I20240426 13:19:53.113829 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.064405 27648 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:53.114048 27496 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.115059 27622 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.369s user 0.002s sys 0.000s
I20240426 13:19:53.115275 24955 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.117058 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.117542 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1960) took 1045 ms. Trace:
I20240426 13:19:53.118360 25027 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.118628 27496 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.112757 27393 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.131491 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.120057 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.123188 25025 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:53.124163 25029 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.123648 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:53.121168 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.138435 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.144904 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1873) took 1051 ms. Trace:
I20240426 13:19:53.145308 24955 rpcz_store.cc:276] 0426 13:19:52.093874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.094039 (+ 165us) service_pool.cc:225] Handling call
0426 13:19:52.124333 (+ 30294us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:53.115645 (+991312us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:53.144692 (+ 29047us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b54000794d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:53.144851 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2278601600}
I20240426 13:19:53.117920 25024 rpcz_store.cc:276] 0426 13:19:52.072494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.079277 (+ 6783us) service_pool.cc:225] Handling call
0426 13:19:52.079580 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.157402 (+ 77822us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.157446 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.173079 (+ 15633us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.464248 (+291169us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:52.464303 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.464392 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.114438 (+650046us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.117237 (+ 2799us) raft_consensus.cc:1671] finished
0426 13:19:53.117258 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.117480 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59656,"raft.queue_time_us":83583,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7519,"spinlock_wait_cycles":7889792,"thread_start_us":253234,"threads_started":1,"wal-append.queue_time_us":551388}
I20240426 13:19:53.133096 24947 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:53.149674 24949 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240426 13:19:53.149807 24948 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.149870 25031 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:53.157016 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.058174 27492 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.157892 24952 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.131525 27393 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:53.162238 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87530.3us
I20240426 13:19:53.163182 25024 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240426 13:19:53.119462 27107 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.168118 27648 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.184103 25102 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240426 13:19:53.146428 27618 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.400s user 0.003s sys 0.001s
I20240426 13:19:53.047518 27498 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.192090 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:19:53.168601 27472 raft_consensus.cc:659] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.169617 24956 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:53.056365 27499 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.066403 27410 raft_consensus.cc:729] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.174909 25029 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:53.181648 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.200495 24995 leader_election.cc:398] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:19:53.200938 24995 leader_election.cc:401] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20240426 13:19:53.185436 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 239127us
W20240426 13:19:53.147258 27327 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.738s user 0.008s sys 0.002s
I20240426 13:19:53.168457 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:19:53.197751 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 71000.5us
I20240426 13:19:53.198120 25096 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:53.247187 25022 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:53.247542 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 115282us
I20240426 13:19:53.180975 27492 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.249317 25099 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:53.203341 24951 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.205443 25026 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20240426 13:19:53.253623 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 53296.8us
I20240426 13:19:53.192481 26824 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.260445 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:19:53.262562 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59261.6us
I20240426 13:19:53.263309 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.263978 27632 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.265514 27631 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.266005 27631 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.267297 25029 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:53.149947 27526 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.281800 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.272470 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.272909 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.300424 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.300802 27379 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.290529 27513 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:53.301716 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.301961 27513 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.302309 25105 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.302932 27513 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.304653 25104 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:53.272542 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.305644 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.306149 25104 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:53.307297 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.308445 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.157447 27376 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.295588 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1961) took 1222 ms (client timeout 1607 ms). Trace:
W20240426 13:19:53.309844 25022 rpcz_store.cc:269] 0426 13:19:52.072879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.076163 (+ 3284us) service_pool.cc:225] Handling call
0426 13:19:53.295326 (+1219163us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001e4e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:53.295505 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586791296}
I20240426 13:19:53.310632 27376 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.309856 25102 raft_consensus.cc:2379] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:53.310731 27638 consensus_queue.cc:1046] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:53.267962 27631 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.312213 27379 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.312605 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.312700 27379 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.313086 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.313802 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.314162 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.313923 27379 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.315166 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.315276 27107 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.316355 27271 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:53.204764 27501 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.317972 27107 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.318529 27107 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.318827 27501 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.315954 25102 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:53.273293 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.286734 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:19:53.290208 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.301178 25097 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:53.322124 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:53.323561 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:53.322373 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 135863us
I20240426 13:19:53.274819 25099 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:19:53.325541 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.326488 24996 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:53.201409 27416 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.308844 27495 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.355103 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.355734 25100 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.347992 25102 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:53.347306 25101 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:53.346976 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98991.4us
I20240426 13:19:53.339100 24952 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:53.333292 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.354581 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:53.363080 24956 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.363034 24948 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.270759 27434 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:53.359863 27150 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.505s user 0.071s sys 0.043s
I20240426 13:19:53.381431 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.381489 25588 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:53.328008 27648 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.286659 27637 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:19:53.347630 27418 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:53.374965 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:53.371048 27416 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:53.385551 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1963) took 1312 ms. Trace:
I20240426 13:19:53.394320 25588 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:53.359670 27379 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.362566 27526 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:53.269197 27498 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.400506 25099 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:53.297201 26679 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.347151 27638 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20240426 13:19:53.409323 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.354501 27631 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240426 13:19:53.402701 25023 rpcz_store.cc:276] 0426 13:19:52.073502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.077732 (+ 4230us) service_pool.cc:225] Handling call
0426 13:19:52.078027 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.078304 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.078327 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.078708 (+ 381us) log.cc:842] Serialized 26 byte log entry
0426 13:19:52.536613 (+457905us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:52.536693 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:52.536825 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.111476 (+574651us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.374464 (+262988us) raft_consensus.cc:1671] finished
0426 13:19:53.374488 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.385478 (+ 10990us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13713280,"thread_start_us":457536,"threads_started":1,"wal-append.queue_time_us":667444}
I20240426 13:19:53.286361 27393 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.327288 26824 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.438486 24948 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.427810 27649 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.408867 27628 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:53.381002 27487 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 13:19:53.409624 27416 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:53.292475 27410 consensus_queue.cc:260] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.442528 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.385318 27438 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20240426 13:19:53.450764 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 89292.1us
I20240426 13:19:53.416589 27647 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.394958 27426 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:19:53.387066 27495 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.399741 27472 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
W20240426 13:19:53.452960 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93564.3us
I20240426 13:19:53.428408 27418 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:53.488063 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46920.6us
I20240426 13:19:53.488905 27418 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.489663 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:53.480834 27655 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240426 13:19:53.481634 25070 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.491262 27495 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.452158 27416 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.460314 27376 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:53.471084 27651 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20240426 13:19:53.485036 27643 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:19:53.486258 27657 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240426 13:19:53.504253 27643 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:53.504724 27643 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.483747 25097 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:53.496382 24951 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.506160 24996 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:53.508143 27249 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:53.508694 27249 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.509544 27249 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.460234 27510 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.498836 25100 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:53.502141 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.442523 27632 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:53.503017 27633 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:53.503372 27640 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:53.588201 27640 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.588652 27631 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.589294 27631 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:53.589004 27640 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.589740 27631 raft_consensus.cc:686] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.590209 27640 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.590658 27631 consensus_queue.cc:237] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.592171 27474 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:53.592530 27474 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:19:53.592870 27659 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.274s user 0.002s sys 0.000s
I20240426 13:19:53.593317 27474 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:53.594344 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164831us
I20240426 13:19:53.474910 27482 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.469144 27393 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:53.559377 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 53347.5us
I20240426 13:19:53.559829 24956 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.417250 27492 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.468806 27649 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:53.515880 25070 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:53.634034 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 131945us
W20240426 13:19:53.637904 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 148330us
I20240426 13:19:53.522416 27510 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.580706 27638 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.577631 27495 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.071s
I20240426 13:19:53.622056 27647 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.671198 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.638698 27474 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.663511 25105 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:53.594964 27633 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:53.640723 27107 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.653860 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:53.665043 27492 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:53.715468 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.684267 24921 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.631347 27482 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.686434 27310 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.660208 27648 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 7 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:53.689365 27327 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.698601 27510 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.713461 25104 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
W20240426 13:19:53.715601 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 96444.9us
I20240426 13:19:53.674638 27496 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:53.725061 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.732771 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.750630 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.715271 27501 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.732148 27496 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.774283 27496 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.730010 27327 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.749828 26824 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:53.778049 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 93850.2us
I20240426 13:19:53.729533 27107 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:53.735661 27310 raft_consensus.cc:2726] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:19:53.723183 27434 consensus_queue.cc:1159] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 7 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:19:53.786585 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 270772us
I20240426 13:19:53.787813 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.725677 27643 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.781281 27376 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:53.797457 27513 raft_consensus.cc:2741] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240426 13:19:53.791936 27107 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.775934 27501 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.810956 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.838956 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:19:53.804553 27647 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:53.814265 25096 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:53.843986 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 52591.5us
I20240426 13:19:53.800695 27628 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:53.797971 27496 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20240426 13:19:53.847100 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 69570.4us
I20240426 13:19:53.810810 27676 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:53.828296 27379 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.840238 27495 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:19:53.855533 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105009us
I20240426 13:19:53.857970 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.849936 27310 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:53.861140 27666 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.861279 27310 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:53.861776 27310 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.861299 27628 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.861735 27666 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.863518 27376 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:53.837683 27633 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.848990 27526 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.916093 27526 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:53.916455 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68016.1us
I20240426 13:19:53.918049 27526 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:53.919135 27645 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.448s user 0.005s sys 0.002s
I20240426 13:19:53.919708 27482 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:53.930972 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 576999us
I20240426 13:19:53.931530 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1888) took 1317 ms. Trace:
I20240426 13:19:53.901058 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.920217 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.920374 26679 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:53.921687 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.922135 27643 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:53.940452 25104 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:19:53.922391 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.923143 27327 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.923059 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.944527 25025 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:53.925442 27416 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.926545 25027 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:53.901623 27379 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.950863 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.952067 27379 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.953032 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:53.931996 24954 rpcz_store.cc:276] 0426 13:19:52.614071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.614859 (+ 788us) service_pool.cc:225] Handling call
0426 13:19:52.641992 (+ 27133us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:52.644189 (+ 2197us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:52.644227 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:52.654316 (+ 10089us) log.cc:842] Serialized 26 byte log entry
0426 13:19:53.131338 (+477022us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:53.131387 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:53.131457 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:53.918338 (+786881us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:53.931199 (+ 12861us) raft_consensus.cc:1671] finished
0426 13:19:53.931227 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:53.931467 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11602688,"thread_start_us":415535,"threads_started":1,"wal-append.queue_time_us":733308}
I20240426 13:19:53.956449 24996 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:53.957145 27501 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.957381 24996 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:53.892992 27651 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20240426 13:19:53.927230 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.901295 27676 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:53.927891 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:53.928436 27675 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240426 13:19:53.929064 27451 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:53.893949 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:53.931706 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:53.850164 27513 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:53.942169 25030 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.945763 25026 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:53.943921 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:53.947453 25104 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:53.940155 27482 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:53.899361 27310 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.948194 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:19:53.912535 27669 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20240426 13:19:53.863816 27666 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.951736 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:53.970572 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.988546 25022 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:53.988829 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94944.3us
I20240426 13:19:53.989177 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:53.989950 25027 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:53.990352 27501 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:53.990876 27501 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:53.991921 25031 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.992550 25097 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:53.991626 27501 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:53.998404 27513 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:53.999614 25105 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:54.001363 27513 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:53.986562 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:53.870064 27674 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
I20240426 13:19:54.005743 27677 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:54.007536 27651 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.008229 27651 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.009096 25022 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:54.010756 27651 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:53.853971 27684 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.011799 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.012392 25023 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:54.012478 24951 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:53.985505 27451 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:53.962679 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 111342us
I20240426 13:19:54.014778 27451 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.022279 25031 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.025945 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:19:54.026837 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.027571 25100 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.027727 27674 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.029995 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.033366 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.030696 27674 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.042276 25588 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:54.043409 27651 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:54.043848 27651 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.044356 27651 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.048471 27327 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:54.051717 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.052425 24956 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:54.053303 27327 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:19:54.055509 26996 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:54.056005 26996 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:54.055596 27674 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.057112 26996 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.057451 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:54.073537 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 141908us
I20240426 13:19:53.949153 27633 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.075361 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.076289 24951 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.078404 25030 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.078795 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.079427 25097 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:19:54.076838 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90362.5us
I20240426 13:19:54.080579 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.081261 24948 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.081466 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.074317 25069 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:54.084018 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.084590 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.084708 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:54.085278 24949 raft_consensus.cc:2379] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.090390 24956 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:53.970342 27107 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:19:53.968657 27676 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.059666 27474 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:53.987463 27438 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.091167 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.139411 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.086357 27495 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.144100 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.088312 27492 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.078034 27647 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.084017 27327 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.152730 24954 raft_consensus.cc:2396] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:19:54.142560 24949 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.152972 24948 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:54.154244 25029 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.102135 27684 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.156989 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.163574 25025 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:54.166553 25030 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:54.177188 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 72958.4us
W20240426 13:19:54.179297 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 105051us
I20240426 13:19:54.189218 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.189293 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:54.181387 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.088467 27666 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.149801 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:54.199229 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 117801us
I20240426 13:19:54.187364 24956 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:54.144320 27676 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.190434 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.206966 25070 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.190918 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.151877 27310 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.209043 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.209148 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.204444 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.212218 24955 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.147164 27107 raft_consensus.cc:686] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.173192 27093 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.194227 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.208910 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.224061 25101 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.205984 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.224543 24947 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:54.174582 27438 raft_consensus.cc:686] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.180160 27327 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.217836 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.230882 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.231246 25069 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:54.231794 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.232583 24950 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:54.235009 24949 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.240690 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.241549 25101 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:54.245599 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.246439 25029 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:19:54.252321 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.253043 25623 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:54.205166 27513 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:54.254324 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.256997 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.183804 27495 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:54.289922 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 58759.1us
I20240426 13:19:54.290377 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.291767 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.292590 25030 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.258052 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:54.295948 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:54.297657 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 90748.1us
I20240426 13:19:54.298139 24956 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:19:54.298529 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.300948 25070 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.242414 27327 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.187322 27647 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.253090 27418 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.302811 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.240414 27682 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.219760 27651 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.309710 25070 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.276631 27666 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.322197 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.304401 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.323315 25101 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 5: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:54.234517 27093 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.295382 27513 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.329331 24948 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:54.330603 24947 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:54.295814 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.306866 27643 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.307087 27303 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.306177 27379 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:54.320006 27434 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.335304 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.306949 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.336580 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:54.307725 25616 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:54.333933 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.345229 24956 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.319733 27416 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.224658 27684 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:54.344851 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 155749us
I20240426 13:19:54.348320 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.364892 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:54.355496 25097 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.357383 25619 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:54.360920 25030 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:54.362211 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.362622 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.367488 24950 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:54.337502 27492 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:54.302204 27687 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20240426 13:19:54.369779 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.377558 24947 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.374405 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.375587 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.383489 24955 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:54.385490 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.387308 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.388942 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.391057 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.400584 25101 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.408459 24953 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:54.412611 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 105742us
I20240426 13:19:54.416307 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.417315 27093 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:54.417752 25621 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:19:54.417827 27482 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.393627 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.438441 24950 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:19:54.439225 24950 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:54.447273 24947 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.379596 27656 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:54.419821 25101 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20240426 13:19:54.420749 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.455456 25097 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:54.422569 24956 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:19:54.416852 27666 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.412006 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.402767 27631 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.375816 27416 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.375674 27501 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:54.397404 27677 raft_consensus.cc:2741] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20240426 13:19:54.418116 27680 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
I20240426 13:19:54.413101 27682 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.452086 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.470808 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.502209 25029 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:54.472201 27496 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.424331 27685 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:54.501272 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.342025 27107 consensus_queue.cc:237] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.449331 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.462839 27482 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.451431 27418 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.477330 24950 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.518435 24950 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:54.508390 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 113609us
I20240426 13:19:54.506626 25622 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:54.346459 27438 consensus_queue.cc:237] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.545537 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:54.538466 27271 raft_consensus.cc:1054] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Unable to promote non-voter 1590f80bdade41c6a755b8501c55fe16: Illegal state: Leader has not yet committed an operation in its own term
I20240426 13:19:54.501832 27682 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.538628 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:54.501000 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:54.505931 27303 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.540849 27416 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.517197 27685 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.557404 24950 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:54.569697 24950 pending_rounds.cc:76] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:54.570060 24950 pending_rounds.cc:97] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:19:54.570412 24950 raft_consensus.cc:2866] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:54.559093 25029 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20240426 13:19:54.561736 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 109787us
I20240426 13:19:54.572528 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.574111 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.517442 27677 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:54.543812 27472 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.468463 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:19:54.545881 27486 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:54.542709 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
W20240426 13:19:54.584465 27692 log.cc:924] Time spent T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20240426 13:19:54.585003 27647 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.501305 27631 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.588013 27651 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.588114 27631 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.589323 27684 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:54.590183 27689 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20240426 13:19:54.590417 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.591795 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:54.592605 25101 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:19:54.562533 27682 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.599390 27303 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.543284 27513 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.558086 27638 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.609004 25024 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:54.550140 27379 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.580289 27310 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.612788 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:54.613612 27701 log.cc:924] Time spent T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20240426 13:19:54.569402 27495 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:54.616758 25622 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:54.542572 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.638657 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:54.615948 27496 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:54.674755 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 204023us
I20240426 13:19:54.663820 27434 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:19:54.661480 27645 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.080s user 0.001s sys 0.003s
I20240426 13:19:54.604271 27327 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.684933 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:54.694511 25102 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.665436 27495 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:54.666146 27643 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.603849 27677 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:54.701115 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 241718us
I20240426 13:19:54.664932 27492 raft_consensus.cc:686] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:54.717695 25030 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:54.721545 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.729176 24950 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:54.728886 25030 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:54.703075 27379 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.682421 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:54.733378 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.750285 24954 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.735188 25102 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:19:54.732928 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 321029us
I20240426 13:19:54.734397 25622 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:19:54.738163 27410 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.737735 27643 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.661617 27418 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.740082 27696 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:54.741536 25623 raft_consensus.cc:2354] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:54.731444 25030 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:54.779742 25030 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:54.699177 27495 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.766317 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:54.782122 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.783439 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:54.790081 25022 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:19:54.789912 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:54.791478 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 217449us
I20240426 13:19:54.740748 27697 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.703194 27434 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:54.778044 27696 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:54.800071 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.784456 27495 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.803427 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:54.826817 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92474.3us
I20240426 13:19:54.792615 27416 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.790009 27107 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:54.857389 27656 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:54.781651 27633 consensus_queue.cc:1159] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" responder_term: 6 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240426 13:19:54.740511 27709 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.175s
I20240426 13:19:54.832429 27684 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:19:54.858857 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.870604 27656 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:54.837028 27482 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:54.874718 25589 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:54.876657 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 76800.5us
I20240426 13:19:54.881237 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.882416 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.883534 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.885710 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 598) took 1272 ms. Trace:
I20240426 13:19:54.878463 27696 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:54.864872 27107 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:54.859016 27303 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.894490 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:54.895428 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:54.886150 25615 rpcz_store.cc:276] 0426 13:19:53.612923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.613184 (+ 261us) service_pool.cc:225] Handling call
0426 13:19:54.885039 (+1271855us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:54.885218 (+ 179us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:54.885391 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.885415 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:54.885458 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:54.885502 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.885557 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:54.885658 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6044416}
I20240426 13:19:54.784353 27715 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20240426 13:19:54.891119 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.895740 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:54.896057 24950 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.875218 27416 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.900502 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:54.863834 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20240426 13:19:54.863740 27631 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20240426 13:19:54.901825 27730 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.902942 27696 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:54.906261 27482 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:54.893083 27501 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:54.908859 24951 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:19:54.943516 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:54.901089 27704 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.231s user 0.001s sys 0.000s
W20240426 13:19:54.945832 27474 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.787s user 0.010s sys 0.003s
I20240426 13:19:54.881294 27685 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:54.896443 27107 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:54.895917 27697 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:54.896895 27684 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:54.995852 27310 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:54.869308 27492 consensus_queue.cc:237] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.034611 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 159984us
I20240426 13:19:55.038236 24950 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:55.013206 27327 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.016482 27496 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:55.052119 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:55.057199 25101 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 7
I20240426 13:19:54.946708 27416 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.028016 27474 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:54.938704 27303 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.048444 27685 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:55.049374 27697 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:19:55.050755 27651 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:19:55.141578 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 71540.1us
I20240426 13:19:55.203116 27416 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.202244 27733 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20240426 13:19:55.039531 27684 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.149003 27631 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
W20240426 13:19:55.209909 27702 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.521s user 0.000s sys 0.000s
W20240426 13:19:55.212550 27662 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 1.744s user 0.007s sys 0.004s
I20240426 13:19:55.203397 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 897) took 1365 ms. Trace:
W20240426 13:19:55.140801 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 197381us
I20240426 13:19:55.210565 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 361) took 1186 ms. Trace:
I20240426 13:19:55.066154 27677 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.148535 27704 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240426 13:19:55.033715 27687 consensus_queue.cc:1046] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20240426 13:19:55.202968 27495 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.262550 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:55.263662 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 1991) took 1326 ms. Trace:
I20240426 13:19:55.264226 27495 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:55.196645 27486 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.593s user 0.008s sys 0.003s
I20240426 13:19:55.199878 27709 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.211230 27688 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.847s user 0.000s sys 0.001s
I20240426 13:19:55.265403 25103 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:55.265002 27709 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.040241 27715 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.266023 27715 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.267048 27709 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.264150 25026 rpcz_store.cc:276] 0426 13:19:53.937318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.000506 (+ 63188us) service_pool.cc:225] Handling call
0426 13:19:54.000901 (+ 395us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.001312 (+ 411us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:19:54.224385 (+223073us) log.cc:842] Serialized 14 byte log entry
0426 13:19:54.618206 (+393821us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.707816 (+ 89610us) log.cc:842] Serialized 26 byte log entry
0426 13:19:54.707988 (+ 172us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:54.708050 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.708135 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.263377 (+555242us) raft_consensus.cc:1671] finished
0426 13:19:55.263408 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.263593 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7605,"prepare.queue_time_us":251418,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":6640,"raft.queue_time_us":57448,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":10603,"thread_start_us":537856,"threads_started":3,"wal-append.queue_time_us":305534}
I20240426 13:19:55.267983 27715 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.207325 27682 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.614s user 0.010s sys 0.000s
I20240426 13:19:55.056249 27699 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20240426 13:19:55.019945 27482 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.268708 27682 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.270248 27682 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.270453 27482 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.067844 27727 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240426 13:19:55.256323 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.268424 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:55.262256 25027 rpcz_store.cc:276] 0426 13:19:54.024148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.588264 (+564116us) service_pool.cc:225] Handling call
0426 13:19:54.588615 (+ 351us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:54.588951 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.588978 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:54.589511 (+ 533us) log.cc:842] Serialized 52 byte log entry
0426 13:19:54.797702 (+208191us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:54.851276 (+ 53574us) log.cc:842] Serialized 14 byte log entry
0426 13:19:54.868150 (+ 16874us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.868249 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.210364 (+342115us) raft_consensus.cc:1671] finished
0426 13:19:55.210384 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.210513 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7177,"prepare.queue_time_us":54417,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":45171,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"spinlock_wait_cycles":189312,"thread_start_us":217013,"threads_started":2,"wal-append.queue_time_us":618266}
I20240426 13:19:55.071682 27740 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.137s
I20240426 13:19:55.273394 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.274201 25026 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:19:55.274089 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.261715 25028 rpcz_store.cc:276] 0426 13:19:53.838313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.842086 (+ 3773us) service_pool.cc:225] Handling call
0426 13:19:55.202798 (+1360712us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.203308 (+ 510us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":367993216}
W20240426 13:19:55.279728 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 270653us
I20240426 13:19:55.280015 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:55.255793 27651 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:55.287626 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 257494us
I20240426 13:19:55.275540 27498 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.260464 27697 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.292668 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.281138 27699 consensus_queue.cc:1225] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 1590f80bdade41c6a755b8501c55fe16 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240426 13:19:55.286566 27685 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.285996 27495 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:55.299432 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.323010 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.325366 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.313349 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.287209 27651 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:55.315240 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:55.294314 27687 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.336478 24954 consensus_meta.cc:287] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.557s user 0.009s sys 0.000s
I20240426 13:19:55.336746 24947 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:55.336815 24954 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:55.294307 27647 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.337776 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:55.294435 27730 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.335134 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.304323 27715 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.322815 27710 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:55.344627 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46741.7us
I20240426 13:19:55.347352 24953 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:55.347652 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 404200us
I20240426 13:19:55.344123 25028 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:55.325141 27510 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:55.351615 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2002) took 1015 ms. Trace:
I20240426 13:19:55.353935 25028 rpcz_store.cc:276] 0426 13:19:54.335807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.313732 (+977925us) service_pool.cc:225] Handling call
0426 13:19:55.335999 (+ 22267us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.351390 (+ 15391us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.351565 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:55.291566 27746 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.200s
I20240426 13:19:55.336300 27482 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:55.329442 27740 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:55.357817 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:55.364384 25619 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:55.342808 27682 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.294955 27498 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.348937 27472 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:55.409425 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 364) took 1251 ms. Trace:
I20240426 13:19:55.398867 27495 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.411494 25027 rpcz_store.cc:276] 0426 13:19:54.158004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.272365 (+1114361us) service_pool.cc:225] Handling call
0426 13:19:55.272741 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.272986 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:19:55.273336 (+ 350us) log.cc:842] Serialized 14 byte log entry
0426 13:19:55.409096 (+135760us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.409147 (+ 51us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:55.409194 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.409258 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.409374 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":170,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":25655,"spinlock_wait_cycles":135936,"thread_start_us":109901,"threads_started":1}
I20240426 13:19:55.418892 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:55.421945 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65620.9us
I20240426 13:19:55.422645 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:55.429930 24954 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:55.350363 27677 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
W20240426 13:19:55.434264 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61914.2us
I20240426 13:19:55.341925 27647 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.428998 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:55.429250 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:55.435096 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:55.431876 25027 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:19:55.412098 27379 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.438040 25588 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:55.412786 27472 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:55.455802 25027 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:55.458063 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.416728 27685 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.464605 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:55.423089 27526 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.473881 25101 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:55.343267 27687 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.412529 27486 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20240426 13:19:55.471601 25027 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.474293 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.356618 27710 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.492043 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58769.3us
I20240426 13:19:55.495756 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:55.498486 25619 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:55.493773 24954 raft_consensus.cc:2379] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:55.499542 24920 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:55.502951 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:55.456156 27715 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.474439 27472 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:55.526899 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:55.502712 27712 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s user 0.001s sys 0.000s
I20240426 13:19:55.490116 27682 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:55.494705 27303 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.470934 27699 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20240426 13:19:55.420372 27740 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.492596 27482 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.519575 27416 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.519856 27738 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:19:55.545974 27684 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.524693 27474 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:55.494421 27526 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.568244 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98116.8us
I20240426 13:19:55.548002 27715 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:55.550153 27725 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.568812 27738 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.603042 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1631) took 1659 ms. Trace:
I20240426 13:19:55.602892 27738 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.604120 27738 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:55.570848 27416 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:19:55.579033 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66426.7us
I20240426 13:19:55.559818 27742 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.437130 27697 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.584245 27486 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.580127 27749 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.582154 27735 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:55.579762 27739 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240426 13:19:55.585393 27745 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
I20240426 13:19:55.581543 27757 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20240426 13:19:55.582593 27765 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:55.570082 27482 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.594659 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.593365 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 445) took 2180 ms. Trace:
W20240426 13:19:55.598778 27744 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.280s user 0.001s sys 0.000s
W20240426 13:19:55.600852 27107 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.555s user 0.015s sys 0.000s
I20240426 13:19:55.601554 27725 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.588376 27474 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:55.605583 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 106087us
I20240426 13:19:55.603516 25105 rpcz_store.cc:276] 0426 13:19:53.943151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.943726 (+ 575us) service_pool.cc:225] Handling call
0426 13:19:53.944053 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:54.000048 (+ 55995us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.602768 (+1602720us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b54003e12d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:55.602951 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127107200}
I20240426 13:19:55.617273 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:55.605624 24995 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.538s (SENT) [suppressed 2 similar messages]
I20240426 13:19:55.586591 27687 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.612515 27758 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20240426 13:19:55.628764 27773 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:55.490259 27379 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.625531 27742 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.626885 27749 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.634083 27765 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:55.640812 27740 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.640385 27771 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.639518 27759 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:19:55.635782 27734 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.446s user 0.003s sys 0.000s
W20240426 13:19:55.625881 27676 consensus_meta.cc:287] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.392s user 0.015s sys 0.006s
I20240426 13:19:55.638181 25617 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:55.629428 27735 raft_consensus.cc:686] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:55.647042 27647 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.639295 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 561) took 1330 ms. Trace:
I20240426 13:19:55.641825 27715 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:19:55.645387 27416 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.640636 27770 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:55.650357 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:55.651708 24952 rpcz_store.cc:276] 0426 13:19:53.413350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.413564 (+ 214us) service_pool.cc:225] Handling call
0426 13:19:53.413919 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:53.490069 (+ 76150us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:53.490098 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:53.490138 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:53.490167 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.591886 (+2101719us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:55.593193 (+ 1307us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.593309 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174725376}
I20240426 13:19:55.564507 27495 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:55.656891 27107 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.657337 27482 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:55.659078 27350 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:19:55.661873 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 360) took 1640 ms. Trace:
W20240426 13:19:55.647572 27666 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.007s sys 0.002s
I20240426 13:19:55.677635 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:55.627534 27486 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.647346 27651 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:55.651535 27730 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20240426 13:19:55.679694 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.960275 27742 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.962039 25103 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1448) took 3465 ms (client timeout 1788 ms). Trace:
W20240426 13:19:55.962852 24948 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1580) took 1753 ms (client timeout 1538 ms). Trace:
W20240426 13:19:55.965534 27310 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.921s user 0.008s sys 0.002s
I20240426 13:19:55.966531 27486 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:55.972887 27526 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.975095 27474 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:55.976300 27696 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.956s user 0.011s sys 0.005s
I20240426 13:19:55.977473 24954 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:19:55.755486 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 234669us
I20240426 13:19:55.993907 25029 rpcz_store.cc:276] 0426 13:19:54.021674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.577809 (+556135us) service_pool.cc:225] Handling call
0426 13:19:54.609019 (+ 31210us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.609229 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.609256 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.620417 (+ 11161us) log.cc:842] Serialized 26 byte log entry
0426 13:19:54.998157 (+377740us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:54.998204 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:54.998288 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:55.661591 (+663303us) raft_consensus.cc:1671] finished
0426 13:19:55.661619 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:55.661804 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":306647,"threads_started":1,"wal-append.queue_time_us":619015}
W20240426 13:19:55.990417 25103 rpcz_store.cc:269] 0426 13:19:52.496864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:52.496980 (+ 116us) service_pool.cc:225] Handling call
0426 13:19:55.961751 (+3464771us) spinlock_profiling.cc:243] Waited 2.75 s on lock 0x7b540009ddd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.961937 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6324418432}
W20240426 13:19:55.991140 24948 rpcz_store.cc:269] 0426 13:19:54.209870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.210094 (+ 224us) service_pool.cc:225] Handling call
0426 13:19:55.962675 (+1752581us) spinlock_profiling.cc:243] Waited 90.2 ms on lock 0x7b5400387a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.962770 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207499008}
W20240426 13:19:55.954639 27510 consensus_meta.cc:287] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.597s user 0.008s sys 0.000s
W20240426 13:19:55.989504 27666 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s user 0.011s sys 0.006s
I20240426 13:19:55.727453 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:55.685314 25099 raft_consensus.cc:1261] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240426 13:19:56.100690 25031 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
W20240426 13:19:56.101153 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 58967.5us
I20240426 13:19:56.102267 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.102587 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:56.103369 25029 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.041924 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.030839 25623 rpcz_store.cc:276] 0426 13:19:54.309089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.309337 (+ 248us) service_pool.cc:225] Handling call
0426 13:19:55.639081 (+1329744us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54004a15d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:55.639217 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":928334208}
I20240426 13:19:56.009349 27767 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.146291 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1992) took 2208 ms (client timeout 2208 ms). Trace:
I20240426 13:19:55.954456 27379 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:55.989758 27716 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.010042 27725 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:19:56.146705 25023 rpcz_store.cc:269] 0426 13:19:53.937587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.950636 (+ 13049us) service_pool.cc:225] Handling call
0426 13:19:56.146074 (+2195438us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b5400055ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.146224 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139899776}
W20240426 13:19:56.134904 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 529281us
I20240426 13:19:56.163630 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.164443 25023 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:56.164705 25105 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:56.166113 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.148262 25022 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240426 13:19:56.066135 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:56.008350 27482 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.153712 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.157402 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.173820 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.173945 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:55.922618 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:56.174417 24955 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:56.007575 27310 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:19:55.989946 27765 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:56.097764 27656 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.203s user 0.008s sys 0.003s
I20240426 13:19:56.168133 25101 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:19:56.101233 27755 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:55.727545 27637 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.174247 25103 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:56.101428 27487 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:56.039369 27770 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:56.167630 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 101629us
W20240426 13:19:56.193763 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.043s sys 0.078s
W20240426 13:19:56.158802 27738 consensus_meta.cc:287] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.554s user 0.009s sys 0.000s
I20240426 13:19:55.735790 27438 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.457s
I20240426 13:19:55.734158 27773 raft_consensus.cc:686] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:56.148101 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 391) took 1917 ms. Trace:
I20240426 13:19:56.178014 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.178354 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.183223 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.184379 25100 raft_consensus.cc:1261] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:56.145838 27416 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.145941 27739 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.189627 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.192096 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 465) took 1469 ms. Trace:
I20240426 13:19:56.234206 24956 rpcz_store.cc:276] 0426 13:19:54.722866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.723033 (+ 167us) service_pool.cc:225] Handling call
0426 13:19:54.764567 (+ 41534us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.793819 (+ 29252us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.793865 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.799457 (+ 5592us) log.cc:842] Serialized 26 byte log entry
0426 13:19:55.015787 (+216330us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:55.015832 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:55.015906 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:56.191816 (+1175910us) raft_consensus.cc:1671] finished
0426 13:19:56.191844 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.192008 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21934,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6187,"thread_start_us":143251,"threads_started":1,"wal-append.queue_time_us":347797}
I20240426 13:19:56.238766 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.148804 27759 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:55.842306 27710 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.098100 27677 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.230824 25023 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:19:56.248054 27677 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.176955 25029 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.249920 27773 consensus_queue.cc:237] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.250790 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:56.194097 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.562s user 0.043s sys 0.078s
I20240426 13:19:56.205857 27755 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.204885 24949 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:56.175554 27715 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.253324 27715 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.254040 27749 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.188616 27697 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.052s sys 0.087s
W20240426 13:19:56.254983 27697 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.052s sys 0.088s
I20240426 13:19:56.193645 27487 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:56.256904 27487 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.257820 27651 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.006201 27771 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.258224 27416 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.225796 24952 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:56.225497 25617 rpcz_store.cc:276] 0426 13:19:54.194709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.204497 (+ 9788us) service_pool.cc:225] Handling call
0426 13:19:55.637784 (+1433287us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:19:55.637999 (+ 215us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.005965 (+367966us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.112009 (+106044us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54551808}
I20240426 13:19:56.188639 27331 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:19:56.251840 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 637046us
I20240426 13:19:56.178263 27482 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.161828 27740 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.156505 27767 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.189121 27735 consensus_queue.cc:237] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.166424 27716 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.351769 24947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.022s sys 0.045s
W20240426 13:19:56.357112 24947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.022s sys 0.046s
I20240426 13:19:56.337257 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 366) took 1661 ms. Trace:
I20240426 13:19:56.338034 24952 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240426 13:19:56.200712 27770 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.358119 25028 rpcz_store.cc:276] 0426 13:19:54.676256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.425812 (+749556us) service_pool.cc:225] Handling call
0426 13:19:55.447608 (+ 21796us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.595545 (+147937us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.595596 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.595647 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:55.595687 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.300287 (+704600us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003af2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:56.337073 (+ 36786us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.337198 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":292663168}
I20240426 13:19:56.270215 24948 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:56.369401 24952 pending_rounds.cc:76] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:19:56.375573 24952 pending_rounds.cc:97] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240426 13:19:56.375931 24952 raft_consensus.cc:2866] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.378198 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.378871 24948 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.370869 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:56.336192 27698 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.887s user 0.009s sys 0.005s
I20240426 13:19:56.368913 27331 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.372361 27771 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.372293 27310 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.373039 27730 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.371093 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.381381 25029 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:56.266337 24954 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:19:56.098434 27676 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.397176 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2006) took 1608 ms. Trace:
I20240426 13:19:56.397545 25023 rpcz_store.cc:276] 0426 13:19:54.788856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.396393 (+1607537us) service_pool.cc:225] Handling call
0426 13:19:56.396691 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.396862 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:56.396888 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.396928 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:56.396956 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.397014 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.397122 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.398442 25029 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:19:56.262395 27438 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:56.260752 27416 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.202333 27656 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.399037 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2005) took 1614 ms. Trace:
I20240426 13:19:56.408140 25023 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:56.408771 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2007) took 1573 ms. Trace:
I20240426 13:19:56.408830 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.369400 27774 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.262073 27778 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.409198 25023 rpcz_store.cc:276] 0426 13:19:54.835084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.404626 (+1569542us) service_pool.cc:225] Handling call
0426 13:19:56.404947 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.408559 (+ 3612us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.408718 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.407608 25029 rpcz_store.cc:276] 0426 13:19:54.785158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.381968 (+1596810us) service_pool.cc:225] Handling call
0426 13:19:56.398154 (+ 16186us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.398807 (+ 653us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.398988 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.411504 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.360194 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:56.412441 25025 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:56.335330 27107 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.423275 24948 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:56.403012 27749 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.398514 27331 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.425801 25102 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:56.204960 27637 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.414498 27416 consensus_queue.cc:1035] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20240426 13:19:56.481016 27771 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20240426 13:19:56.488283 27749 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:19:56.489794 27676 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.263543 27739 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.475551 27656 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.473624 26996 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.491835 27739 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.400249 27310 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.492565 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.044s sys 0.053s
W20240426 13:19:56.492887 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.044s sys 0.053s
I20240426 13:19:56.493661 27310 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.494180 27310 raft_consensus.cc:2726] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.494791 27310 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:19:56.482314 27673 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.485783 25101 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:56.377403 27740 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:56.480249 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 62815.2us
I20240426 13:19:56.467909 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.502378 24952 raft_consensus.cc:1261] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:19:56.430390 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.399741 27647 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.416910 27677 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.491747 27749 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.495747 27755 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.425308 27107 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.526538 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 115110us
I20240426 13:19:56.406486 27697 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20240426 13:19:56.407086 27730 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.528096 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.520700 27715 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.542371 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 305373us
I20240426 13:19:56.546119 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.546592 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:19:56.550915 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 384865us
I20240426 13:19:56.551720 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:56.552398 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 427804us
I20240426 13:19:56.425904 27774 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.558821 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.520362 27647 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.562675 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.566843 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.567485 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:56.567601 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.417542 27778 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.544431 27749 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:56.561586 27472 consensus_meta.cc:287] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.995s user 0.011s sys 0.000s
I20240426 13:19:56.518168 27673 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.547329 27438 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:56.563835 27697 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.530869 27107 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.561774 27755 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.557786 27418 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:56.578013 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.001s sys 0.022s
I20240426 13:19:56.566792 27310 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:19:56.582564 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.589378 25103 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:56.583048 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 81152us
I20240426 13:19:56.584118 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:56.585805 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.001s sys 0.022s
I20240426 13:19:56.592944 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.591781 25588 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:19:56.596387 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.596822 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:56.499380 27759 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.603184 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.608004 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.611196 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.611850 24948 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:19:56.611745 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:56.613902 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 45479.3us
I20240426 13:19:56.612412 24952 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:56.617942 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.618243 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.618826 24952 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:56.581107 27472 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.585889 27379 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.625499 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.634800 24948 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:56.634925 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.582731 27482 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.627470 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.589802 27793 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.636409 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.638913 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.608325 27438 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:56.605800 27677 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:56.646260 24952 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.647894 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.648473 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.648869 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.649405 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2008) took 1637 ms. Trace:
I20240426 13:19:56.652207 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.650815 25023 rpcz_store.cc:276] 0426 13:19:55.012130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.418429 (+1406299us) service_pool.cc:225] Handling call
0426 13:19:56.418733 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.419068 (+ 335us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:56.579767 (+160699us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.649100 (+ 69333us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.649153 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:56.649197 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.649260 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.649358 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":216709,"thread_start_us":152816,"threads_started":2}
I20240426 13:19:56.654636 25620 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:56.650033 24948 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.578732 27685 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20240426 13:19:56.625456 27716 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.655735 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.658926 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.572669 27788 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20240426 13:19:56.590518 27755 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.662053 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.662860 24948 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:56.594861 27418 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.662987 25105 raft_consensus.cc:2396] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:56.664616 25023 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:56.665619 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.667660 24948 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:56.666544 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.605572 27749 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240426 13:19:56.671638 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:56.609616 27676 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20240426 13:19:56.672863 25023 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:56.661093 27783 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.236s user 0.000s sys 0.000s
I20240426 13:19:56.657641 27482 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:56.647315 27438 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.680685 27796 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:56.681011 27438 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:56.681337 27796 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.681835 27796 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:56.680735 27482 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.685137 27482 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.657687 27715 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:56.679256 27755 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.688392 25023 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:19:56.656806 27495 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:19:56.651237 27677 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.689493 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2010) took 1325 ms. Trace:
I20240426 13:19:56.691483 25023 rpcz_store.cc:276] 0426 13:19:55.364579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.687683 (+1323104us) service_pool.cc:225] Handling call
0426 13:19:56.687976 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.689155 (+ 1179us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.689429 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.615381 27794 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20240426 13:19:56.658164 27767 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.675721 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.711900 27801 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:56.712474 27801 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:56.715618 27786 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20240426 13:19:56.716826 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 476) took 1051 ms. Trace:
I20240426 13:19:56.717151 27482 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.717209 24947 rpcz_store.cc:276] 0426 13:19:55.665605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.665776 (+ 171us) service_pool.cc:225] Handling call
0426 13:19:55.667018 (+ 1242us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.667340 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.667370 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.674216 (+ 6846us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.357685 (+683469us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:56.357740 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.357833 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:56.716599 (+358766us) raft_consensus.cc:1671] finished
0426 13:19:56.716618 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.716770 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57693,"raft.queue_time_us":939,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":45750,"thread_start_us":583209,"threads_started":1,"wal-append.queue_time_us":836096}
I20240426 13:19:56.718866 27451 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:56.719174 24955 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20240426 13:19:56.719326 27482 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.721109 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1589) took 1407 ms. Trace:
I20240426 13:19:56.721483 24953 rpcz_store.cc:276] 0426 13:19:55.313258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.314100 (+ 842us) service_pool.cc:225] Handling call
0426 13:19:56.720904 (+1406804us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b54002ca6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.721050 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14568832}
I20240426 13:19:56.722667 27774 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:56.724529 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 1987) took 2798 ms (client timeout 1686 ms). Trace:
W20240426 13:19:56.724999 25024 rpcz_store.cc:269] 0426 13:19:53.925774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.927632 (+ 1858us) service_pool.cc:225] Handling call
0426 13:19:56.724316 (+2796684us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.724460 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1404053760}
I20240426 13:19:56.713318 27801 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.728888 25027 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.734781 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2011) took 1315 ms. Trace:
I20240426 13:19:56.735298 25023 rpcz_store.cc:276] 0426 13:19:55.419443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.712105 (+1292662us) service_pool.cc:225] Handling call
0426 13:19:56.712401 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.734330 (+ 21929us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:56.734375 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.734433 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:19:56.734479 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.734571 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.734714 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21571}
I20240426 13:19:56.664999 27740 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.668720 27716 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:56.648203 27379 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.742421 25024 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:19:56.747545 27495 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.747720 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.748013 27774 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:56.751034 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 154311us
I20240426 13:19:56.752461 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.754413 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 370) took 1317 ms. Trace:
I20240426 13:19:56.677505 27637 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.754920 25024 rpcz_store.cc:276] 0426 13:19:55.437179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.725784 (+1288605us) service_pool.cc:225] Handling call
0426 13:19:56.726139 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.754093 (+ 27954us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.754344 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:56.694259 27677 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:56.758625 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:56.759581 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:56.761152 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.764707 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.766722 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.768474 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:56.748845 27716 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.787951 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.785347 27767 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:56.790035 27767 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.790513 27767 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:56.790287 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.749830 27730 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.796680 27794 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.784732 27637 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.798292 27637 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.799154 27331 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:56.801381 27732 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.660s user 0.006s sys 0.000s
I20240426 13:19:56.797363 27794 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.817004 27794 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.817705 27637 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.818904 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:19:56.821118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61607.4us
I20240426 13:19:56.803225 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.775779 25027 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20240426 13:19:56.821213 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:56.820539 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 193144us
I20240426 13:19:56.822577 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.814787 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:19:56.823334 25096 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1454) took 3548 ms (client timeout 1737 ms). Trace:
I20240426 13:19:56.823740 25105 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.825492 25027 pending_rounds.cc:76] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 7
I20240426 13:19:56.826046 25027 pending_rounds.cc:97] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20240426 13:19:56.826427 25027 raft_consensus.cc:2866] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.761508 27647 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.651239 27793 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.828598 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.789218 27716 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:56.828871 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.801712 27331 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.849715 24947 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 9
W20240426 13:19:56.823740 25096 rpcz_store.cc:269] 0426 13:19:53.275058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:53.308212 (+ 33154us) service_pool.cc:225] Handling call
0426 13:19:56.823174 (+3514962us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54003eeed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:56.823268 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132486912}
I20240426 13:19:56.850483 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.851310 24948 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.860632 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:56.862403 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 367) took 2181 ms. Trace:
I20240426 13:19:56.822499 27746 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.757156 27495 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.862808 25027 rpcz_store.cc:276] 0426 13:19:54.678726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.431491 (+752765us) service_pool.cc:225] Handling call
0426 13:19:55.431761 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.728537 (+1296776us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b58004c01b0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda902930e3 00007fda902f6c4a 00007fda9030e53f 00007fda9030fcde 00007fda90319e19 00007fda9031a6e6 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654
0426 13:19:56.827181 (+ 98644us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.860007 (+ 32826us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":385102720}
I20240426 13:19:56.873472 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.761114 27774 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:56.877619 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:56.878350 25027 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:19:56.881415 24947 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:56.829407 27150 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:56.883865 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:56.885970 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.888069 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:56.887858 25027 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:19:56.834790 27685 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:56.852589 27673 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.876269 27778 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.875185 27472 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.891101 27150 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:56.892740 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 300939us
I20240426 13:19:56.891671 27804 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:56.895287 27685 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.919744 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.933243 25615 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:19:56.910650 27799 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.905288 27773 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:56.906555 27647 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:56.904649 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.891155 27808 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:19:56.928084 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54723.3us
W20240426 13:19:56.920060 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 151668us
I20240426 13:19:56.935056 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 371) took 1372 ms. Trace:
I20240426 13:19:56.963480 25024 rpcz_store.cc:276] 0426 13:19:55.562486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.755936 (+1193450us) service_pool.cc:225] Handling call
0426 13:19:56.786497 (+ 30561us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.786774 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:19:56.787117 (+ 343us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.934734 (+147617us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.934797 (+ 63us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:56.934841 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.934904 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.935006 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13269,"prepare.queue_time_us":943,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":13373,"thread_start_us":133967,"threads_started":1}
I20240426 13:19:56.964331 24995 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.866315 27107 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:56.949254 25096 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240426 13:19:56.965473 25096 pending_rounds.cc:76] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 1
I20240426 13:19:56.965675 25096 pending_rounds.cc:97] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240426 13:19:56.965905 25096 raft_consensus.cc:2866] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:56.914988 27794 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:56.919338 27498 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:56.885427 27746 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.940122 27482 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:56.934302 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:56.960997 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.959098 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 369) took 1698 ms. Trace:
I20240426 13:19:56.965252 25105 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:56.967806 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:56.926240 27685 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.919179 27150 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:56.965957 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.977523 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:56.950248 27799 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:56.948359 27418 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:56.999080 24949 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 9
I20240426 13:19:56.979702 25029 rpcz_store.cc:276] 0426 13:19:55.261131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.412057 (+1150926us) service_pool.cc:225] Handling call
0426 13:19:56.428138 (+ 16081us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.428338 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:56.901577 (+473239us) log.cc:842] Serialized 14 byte log entry
0426 13:19:56.958751 (+ 57174us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:56.958802 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:56.958845 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.958916 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:56.959043 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67194,"prepare.queue_time_us":243934,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":3616,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":8292,"thread_start_us":462403,"threads_started":2}
I20240426 13:19:56.955296 27327 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:56.978813 27482 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.009547 27482 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:57.002225 25101 raft_consensus.cc:3037] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.002955 27495 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.004022 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.004334 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:56.982739 27730 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240426 13:19:56.968984 27647 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.017172 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:57.002506 24996 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:57.020896 25029 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:19:57.023641 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 57790.3us
I20240426 13:19:57.025341 24949 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:56.975112 27755 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
W20240426 13:19:56.982452 27797 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.271s user 0.001s sys 0.001s
I20240426 13:19:57.029666 24995 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:57.034332 27770 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.634s user 0.010s sys 0.007s
I20240426 13:19:57.041492 27107 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.041936 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:56.957628 27773 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.026708 27740 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:56.962179 27801 consensus_queue.cc:1035] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 13:19:57.020870 27379 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.058553 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:56.974876 27794 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.061496 27793 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.061944 27810 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:57.062443 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:57.063095 27773 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.053755 27496 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.080893 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.062875 27482 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:57.063532 27812 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20240426 13:19:57.063824 25097 raft_consensus.cc:595] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:57.111644 25097 raft_consensus.cc:603] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:19:57.111889 25029 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:19:57.055938 27740 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:56.934737 27804 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:57.090479 27821 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20240426 13:19:57.090585 27799 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:57.105165 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46693us
I20240426 13:19:56.978598 27808 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.064733 27735 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20240426 13:19:57.116633 27709 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:57.050565 27770 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.065096 25588 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
W20240426 13:19:57.064237 27816 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20240426 13:19:57.064064 27438 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.100458 27716 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.110101 25105 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.108044 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1586) took 1797 ms. Trace:
I20240426 13:19:57.113818 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1641) took 1770 ms. Trace:
I20240426 13:19:57.131466 25099 rpcz_store.cc:276] 0426 13:19:55.343728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.343977 (+ 249us) service_pool.cc:225] Handling call
0426 13:19:55.358079 (+ 14102us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.099646 (+741567us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.113597 (+1013951us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400168d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.113765 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":737609088}
I20240426 13:19:57.135457 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:57.137032 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 72006.3us
I20240426 13:19:57.138540 25101 raft_consensus.cc:2454] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:57.115521 27740 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.156410 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.157114 25099 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:19:57.130358 24951 rpcz_store.cc:276] 0426 13:19:55.310711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.311265 (+ 554us) service_pool.cc:225] Handling call
0426 13:19:55.338687 (+ 27422us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.339058 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 1
0426 13:19:55.339096 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.339500 (+ 404us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.728025 (+1388525us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b1800184960. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:56.904935 (+176910us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:56.904972 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.905036 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.105319 (+200283us) raft_consensus.cc:1671] finished
0426 13:19:57.105338 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.107969 (+ 2631us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6008,"raft.queue_time_us":2859,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11973,"spinlock_wait_cycles":4834560,"thread_start_us":147035,"threads_started":1,"wal-append.queue_time_us":1704021}
I20240426 13:19:57.159066 27793 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.158931 27796 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.159675 27793 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.161237 27716 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.161327 27438 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.161692 27793 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.161904 27716 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.111433 27496 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.162317 27810 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:57.162153 27685 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.164009 27496 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.162326 27716 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.164764 27716 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.164956 27379 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.165472 27379 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.167405 27379 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.114894 27768 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:57.107758 27673 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:19:57.170159 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 467) took 3151 ms. Trace:
I20240426 13:19:57.172590 25023 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:19:57.178561 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:57.174032 25098 rpcz_store.cc:276] 0426 13:19:54.018851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.018969 (+ 118us) service_pool.cc:225] Handling call
0426 13:19:54.019330 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.173952 (+154622us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.173996 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.201717 (+ 27721us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.414223 (+2212506us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b1800437c80. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:56.540766 (+126543us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:56.540829 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.540925 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.117878 (+576953us) raft_consensus.cc:1671] finished
0426 13:19:57.117902 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.170086 (+ 52184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164034,"raft.queue_time_us":172246,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9526,"spinlock_wait_cycles":18676480,"thread_start_us":126308,"threads_started":1,"wal-append.queue_time_us":2389752}
W20240426 13:19:57.121651 27823 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240426 13:19:57.186667 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:57.163425 27810 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.189983 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.191205 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.131165 27774 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.155910 27482 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.066164 27755 raft_consensus.cc:686] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:19:57.173033 27735 raft_consensus.cc:2781] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:19:57.202520 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 160678us
I20240426 13:19:57.216606 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.203120 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.173544 27673 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.120465 27709 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.217972 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.219957 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.190407 27801 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.189986 27685 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.182982 27677 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:19:57.226111 27673 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:57.179289 27773 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20240426 13:19:57.223529 27482 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:57.227807 27801 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:57.294616 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74838.6us
I20240426 13:19:57.321404 24995 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.194591 27793 consensus_queue.cc:1035] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240426 13:19:57.226325 27735 raft_consensus.cc:686] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:19:57.312343 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 79028.9us
W20240426 13:19:57.313556 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 84698.6us
I20240426 13:19:57.315322 25101 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.219534 27628 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:57.328305 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.329434 25098 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:19:57.176348 27768 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.295677 27496 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.331594 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:19:57.323307 27786 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s user 0.007s sys 0.000s
I20240426 13:19:57.318475 27438 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.319977 27482 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.331116 27637 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.322472 27107 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:57.323057 27801 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.324950 27495 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.325134 27773 consensus_queue.cc:1225] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:19:57.343684 27496 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.351979 27827 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:19:57.324851 27150 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.348484 27438 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.334297 27794 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.341933 27804 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.348448 27482 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:57.369827 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.359082 27107 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.357589 27827 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:19:57.448237 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126918us
I20240426 13:19:57.449272 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:57.453835 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.454838 25101 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:57.370018 27796 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.353854 27637 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.375114 27793 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:57.357321 27495 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.394219 27770 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.341733 27755 consensus_queue.cc:237] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.463496 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1633) took 3386 ms. Trace:
I20240426 13:19:57.501106 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.439777 27808 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.400799 27827 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:19:57.650285 24952 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:19:57.632629 25104 rpcz_store.cc:276] 0426 13:19:54.077142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.078167 (+ 1025us) service_pool.cc:225] Handling call
0426 13:19:54.086311 (+ 8144us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:54.191143 (+104832us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:54.191170 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:54.191560 (+ 390us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.974425 (+2782865us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b18003f61c0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:19:57.094149 (+119724us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:19:57.094187 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.094270 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.456236 (+361966us) raft_consensus.cc:1671] finished
0426 13:19:57.456258 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.463427 (+ 7169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92698,"raft.queue_time_us":118132,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":8366,"spinlock_wait_cycles":60471680,"thread_start_us":119570,"threads_started":1,"wal-append.queue_time_us":3031559}
I20240426 13:19:57.623661 25030 raft_consensus.cc:1261] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20240426 13:19:57.652288 25030 pending_rounds.cc:76] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:19:57.652585 25030 pending_rounds.cc:97] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20240426 13:19:57.652834 25030 raft_consensus.cc:2866] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:57.654732 27495 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.656783 24956 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20240426 13:19:57.658828 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 472250us
I20240426 13:19:57.653504 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 362) took 3628 ms. Trace:
I20240426 13:19:57.497893 27843 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.439742 27107 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:57.645169 27765 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.437s user 0.012s sys 0.009s
I20240426 13:19:57.674903 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:57.677938 27716 consensus_meta.cc:287] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.513s user 0.009s sys 0.000s
I20240426 13:19:57.678678 27832 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.679215 27832 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.679709 27839 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.321s
I20240426 13:19:57.656551 27474 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.680529 27474 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.456738 27740 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.665869 25101 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:57.667181 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 215783us
I20240426 13:19:57.661149 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.678385 27716 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.659976 27677 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
W20240426 13:19:57.681241 27834 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20240426 13:19:57.691848 27767 consensus_meta.cc:287] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.858s user 0.010s sys 0.001s
I20240426 13:19:57.698031 27150 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.399052 27735 consensus_queue.cc:237] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.695349 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.701673 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.702055 24948 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:57.702306 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.703220 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.703999 24956 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:57.705691 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.706235 24953 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:57.709046 27637 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.713618 27835 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:57.713912 27716 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.714749 27847 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:57.716300 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:57.716475 27350 raft_consensus.cc:1054] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Unable to promote non-voter 1590f80bdade41c6a755b8501c55fe16: Illegal state: Replica 1e87e7bb414642aeb72fe0d2ff72cc2e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } }
I20240426 13:19:57.716921 24951 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:57.718700 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.719345 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.721264 24953 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
W20240426 13:19:57.747113 27831 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.289s user 0.000s sys 0.001s
I20240426 13:19:57.748477 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:57.749217 24951 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:57.754770 27740 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.683754 27765 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.721815 27438 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.737633 27801 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.761844 27832 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.763027 27801 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.671089 27843 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.741389 27150 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.666532 25030 rpcz_store.cc:276] 0426 13:19:54.024802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.692278 (+667476us) service_pool.cc:225] Handling call
0426 13:19:54.717392 (+ 25114us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.622953 (+2905561us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b540006c0a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda9031a057 00007fda9031a740 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:19:57.653254 (+ 30301us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.653452 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18504448}
I20240426 13:19:57.751217 27637 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:57.718681 27767 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.777541 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:57.773165 27150 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.782680 27150 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.779561 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.789528 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:57.791334 27677 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.801640 24951 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:19:57.791832 27677 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.804901 25105 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
W20240426 13:19:57.805899 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 147957us
I20240426 13:19:57.796166 27416 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.773418 27637 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.800303 27832 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:57.811033 24956 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
W20240426 13:19:57.810635 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146036us
I20240426 13:19:57.821491 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.824216 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.829442 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.830273 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:57.831228 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.850503 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:57.807428 27107 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.808517 27843 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.800364 27673 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.888864 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:57.882293 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:57.906994 25096 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:57.890383 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.909507 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:19:57.912961 27827 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:57.916635 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 605) took 2477 ms. Trace:
I20240426 13:19:57.879839 27832 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.916746 27827 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.924188 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.925444 24953 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.917263 25620 rpcz_store.cc:276] 0426 13:19:55.438951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.439178 (+ 227us) service_pool.cc:225] Handling call
0426 13:19:55.471415 (+ 32237us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.661579 (+1190164us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.916349 (+1254770us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54006be9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.916551 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2720682880}
W20240426 13:19:57.841256 27759 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.010s sys 0.003s
I20240426 13:19:57.914356 27496 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.933573 24948 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:57.934676 24951 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:19:57.937506 27640 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:57.937734 27677 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.935798 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1491) took 2311 ms. Trace:
I20240426 13:19:57.943029 25027 rpcz_store.cc:276] 0426 13:19:55.624598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.881606 (+1257008us) service_pool.cc:225] Handling call
0426 13:19:56.883269 (+ 1663us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.892953 (+ 9684us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.935575 (+1042622us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b54000555d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:57.935740 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9804544}
I20240426 13:19:57.944242 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.944855 25027 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:57.944904 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1598) took 1746 ms. Trace:
I20240426 13:19:57.945719 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.948668 27640 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:57.949071 27640 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:19:57.950316 27795 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 1.361s user 0.011s sys 0.000s
I20240426 13:19:57.941031 24949 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:19:57.953081 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.953523 27637 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:57.953790 25027 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:19:57.954120 27765 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.954798 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.955226 27767 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.955564 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:57.956547 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.957407 27673 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:57.957912 27709 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:57.958102 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.959710 25027 raft_consensus.cc:2379] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.945398 24955 rpcz_store.cc:276] 0426 13:19:56.198388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.198531 (+ 143us) service_pool.cc:225] Handling call
0426 13:19:56.718914 (+520383us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.719575 (+ 661us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:57.944673 (+1225098us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540004bd50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:57.944835 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195890048}
I20240426 13:19:57.963639 24947 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.963953 24949 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:19:57.959900 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.966658 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.967263 24948 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:19:57.941715 27759 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.011s sys 0.003s
I20240426 13:19:57.969039 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 401) took 2539 ms. Trace:
I20240426 13:19:57.969550 25616 rpcz_store.cc:276] 0426 13:19:55.429160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.433601 (+ 4441us) service_pool.cc:225] Handling call
0426 13:19:55.454141 (+ 20540us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.787164 (+333023us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:19:55.834023 (+ 46859us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:56.106725 (+272702us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.519065 (+412340us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:56.521715 (+ 2650us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.526781 (+ 5066us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:57.949691 (+1422910us) spinlock_profiling.cc:243] Waited 8.52 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:19:57.961706 (+ 12015us) raft_consensus.cc:1671] finished
0426 13:19:57.963352 (+ 1646us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:57.968974 (+ 5622us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102169,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3997,"spinlock_wait_cycles":19588864,"thread_start_us":512199,"threads_started":2,"wal-append.queue_time_us":379994}
I20240426 13:19:57.968216 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.980139 24948 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:19:57.837769 27416 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:57.961165 27843 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:57.986477 24953 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:57.986609 24949 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20240426 13:19:57.954746 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 399) took 2526 ms (client timeout 2021 ms). Trace:
W20240426 13:19:57.984290 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.001s sys 0.042s
W20240426 13:19:57.987879 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s user 0.002s sys 0.042s
I20240426 13:19:57.984990 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.966363 27677 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:57.996596 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.997387 24949 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:19:57.992247 25027 raft_consensus.cc:2379] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:57.992355 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.000552 25030 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:58.001008 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.001858 24949 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:19:58.004763 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.994761 27677 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.010373 27677 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.012326 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.012876 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:57.995496 27832 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:57.990471 25618 rpcz_store.cc:269] 0426 13:19:55.428529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.428681 (+ 152us) service_pool.cc:225] Handling call
0426 13:19:57.954493 (+2525812us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b54005a2dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:57.954628 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":927631744}
I20240426 13:19:58.009711 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.013062 25030 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:58.018164 24947 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:19:58.020023 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.022825 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.007678 25027 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:19:58.023962 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2015) took 1871 ms. Trace:
I20240426 13:19:58.024389 25027 rpcz_store.cc:276] 0426 13:19:56.152438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.007019 (+1854581us) service_pool.cc:225] Handling call
0426 13:19:58.007301 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.023600 (+ 16299us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.023898 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:58.025506 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:57.995784 27496 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20240426 13:19:58.030336 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.030603 27496 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.031150 27496 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:58.032205 27496 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.038313 27640 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.041118 27416 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:58.044149 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 397) took 3241 ms (client timeout 1746 ms). Trace:
I20240426 13:19:58.044391 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:58.044659 25622 rpcz_store.cc:269] 0426 13:19:54.802955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.803102 (+ 147us) service_pool.cc:225] Handling call
0426 13:19:58.043959 (+3240857us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b5400490a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:58.044071 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4953170816}
I20240426 13:19:58.045025 24951 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.046422 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.043990 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.049237 24955 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:58.049461 27750 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 2.691s user 0.011s sys 0.022s
I20240426 13:19:58.050223 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.050938 24953 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:58.051826 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.054018 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.055537 24949 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:19:58.056979 24955 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.061259 27496 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.063844 27687 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:19:58.064657 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.000151 27843 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:58.078824 27728 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20240426 13:19:58.080447 27640 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:58.086089 27628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.008s sys 0.033s
W20240426 13:19:58.144933 27482 consensus_meta.cc:287] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.770s user 0.011s sys 0.000s
I20240426 13:19:58.145485 27482 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.146883 25027 raft_consensus.cc:2379] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:58.143124 27677 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:58.153053 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.013s sys 0.049s
W20240426 13:19:58.153904 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.013s sys 0.049s
I20240426 13:19:58.151746 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2001) took 3816 ms. Trace:
I20240426 13:19:58.158264 25026 rpcz_store.cc:276] 0426 13:19:54.335531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.307940 (+972409us) service_pool.cc:225] Handling call
0426 13:19:55.344272 (+ 36332us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:55.409880 (+ 65608us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.409949 (+ 69us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:55.410029 (+ 80us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:19:55.410094 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.151419 (+2741325us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b54003af7d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:58.151483 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.151652 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48932096}
W20240426 13:19:58.126669 27628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.008s sys 0.034s
I20240426 13:19:58.147934 27482 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.185106 27628 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:58.225399 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.299743 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.226951 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:58.294690 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.008s sys 0.067s
W20240426 13:19:58.185240 25024 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.080s user 0.020s sys 0.056s
W20240426 13:19:58.210366 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.012s sys 0.012s
W20240426 13:19:58.304235 25070 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.016s sys 0.009s
I20240426 13:19:58.307438 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.329448 25620 raft_consensus.cc:1261] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:19:58.301226 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.308770 25099 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
W20240426 13:19:58.276484 27683 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.483s user 0.018s sys 0.030s
I20240426 13:19:58.315702 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.336350 27796 consensus_queue.cc:1035] T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20240426 13:19:58.330821 25030 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:19:58.299551 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.345538 25026 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:19:58.339488 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.133s user 0.008s sys 0.027s
I20240426 13:19:58.339661 27628 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.319447 27474 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:19:58.331251 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.017s sys 0.013s
W20240426 13:19:58.419620 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 759871us
I20240426 13:19:58.332204 25104 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:19:58.331140 25024 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.226s user 0.020s sys 0.056s
I20240426 13:19:58.330716 24951 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:19:58.330999 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s user 0.009s sys 0.067s
I20240426 13:19:58.426710 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2013) took 2383 ms. Trace:
W20240426 13:19:58.334276 25070 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.537s user 0.018s sys 0.009s
W20240426 13:19:58.347671 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.142s user 0.009s sys 0.027s
W20240426 13:19:58.435634 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 638337us
I20240426 13:19:58.343686 25096 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
W20240426 13:19:58.352643 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 52009us
I20240426 13:19:58.359148 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.583711 25098 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.584636 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:19:58.589020 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 152291us
W20240426 13:19:58.330449 27740 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.013s sys 0.015s
I20240426 13:19:58.595176 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.427223 25024 rpcz_store.cc:276] 0426 13:19:56.043319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.964933 (+921614us) service_pool.cc:225] Handling call
0426 13:19:56.980847 (+ 15914us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.981020 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:19:56.981675 (+ 655us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.422790 (+1441115us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:58.426336 (+ 3546us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:58.426406 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.426497 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.426645 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":824,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":59933,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":1437180,"threads_started":2,"wal-append.queue_time_us":144298,"wal-append.run_cpu_time_us":1017,"wal-append.run_wall_time_us":1001047}
W20240426 13:19:58.299330 27835 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.022s sys 0.021s
I20240426 13:19:58.591815 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.586726 27724 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
W20240426 13:19:58.594702 27740 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.800s user 0.013s sys 0.015s
I20240426 13:19:58.599581 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.594830 27778 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20240426 13:19:58.595677 27833 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.299731 27775 raft_consensus.cc:3032] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.595703 25030 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.654168 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.609483 25621 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:58.662966 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.604256 27843 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.605913 24996 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:58.623268 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.608486 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.625651 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.627213 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.619694 25024 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
W20240426 13:19:58.623502 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.007s sys 0.018s
I20240426 13:19:58.598412 24951 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
W20240426 13:19:58.669801 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 63964.6us
I20240426 13:19:58.617389 27850 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:19:58.624593 25623 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:19:58.599761 25068 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.017s sys 0.027s
I20240426 13:19:58.641865 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:19:58.685433 25068 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.964s user 0.018s sys 0.027s
W20240426 13:19:58.616950 27835 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.022s sys 0.022s
I20240426 13:19:58.683790 27640 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20240426 13:19:58.648634 27775 raft_consensus.cc:729] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:58.617174 27724 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.664856 25104 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
W20240426 13:19:58.585180 27683 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.554s user 0.020s sys 0.032s
I20240426 13:19:58.675979 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2020) took 2470 ms. Trace:
W20240426 13:19:58.672044 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.007s sys 0.018s
I20240426 13:19:58.630342 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.682582 25098 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:58.683698 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.670410 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:58.643747 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.683502 27855 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.688165 27474 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.688352 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.688655 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.611028 26824 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240426 13:19:58.615315 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.698782 24921 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.017s sys 0.022s
W20240426 13:19:58.731770 24921 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.575s user 0.017s sys 0.023s
W20240426 13:19:58.732206 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 575463us
I20240426 13:19:58.700495 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 482) took 3827 ms. Trace:
I20240426 13:19:58.733007 25102 rpcz_store.cc:276] 0426 13:19:54.873604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.873775 (+ 171us) service_pool.cc:225] Handling call
0426 13:19:54.874116 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:56.429195 (+1555079us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.700230 (+2271035us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:58.700429 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3568210816}
W20240426 13:19:58.734836 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 119614us
I20240426 13:19:58.735091 25102 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:19:58.632148 26679 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20240426 13:19:58.702668 25024 rpcz_store.cc:276] 0426 13:19:56.205676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.610267 (+2404591us) service_pool.cc:225] Handling call
0426 13:19:58.610581 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.675681 (+ 65100us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.675918 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:58.704159 25099 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:19:58.683048 27677 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.597714 27482 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:19:58.748756 25098 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:58.749318 25098 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.719691 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.668000 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.668435 27868 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.735746 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.738175 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.775063 26824 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:19:58.785712 27863 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240426 13:19:58.785782 26824 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.788280 27775 consensus_queue.cc:260] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.789256 27647 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:58.789592 27775 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.789901 27647 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:19:58.795162 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.797312 25096 raft_consensus.cc:1261] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 13:19:58.804770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69129us
I20240426 13:19:58.699221 27724 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.769795 27766 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.726605 27855 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.777172 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.810731 25024 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.810914 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.812495 27740 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:58.812927 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.814100 27740 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:58.815676 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:58.763022 27850 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:19:58.762544 27474 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.778188 27778 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.778843 27496 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:58.778905 27796 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:58.779578 27852 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.780696 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.770925 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:58.781914 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:58.750264 25098 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.819952 25099 raft_consensus.cc:2396] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:19:58.820370 25098 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:58.805166 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:58.795948 25104 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 9
I20240426 13:19:58.806879 26824 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:19:58.757755 25030 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:19:58.850081 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.795163 27868 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:19:58.829025 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 47206.5us
I20240426 13:19:58.806541 27862 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240426 13:19:58.818209 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 164915us
I20240426 13:19:58.861966 25096 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:19:58.816819 27855 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.818866 27724 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.867396 27868 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.859072 27862 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:58.864763 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:58.819664 27677 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:58.851583 27850 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:58.816262 27766 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.883656 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.889165 25030 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:58.893134 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:58.893708 25030 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:19:58.897449 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2014) took 2789 ms. Trace:
I20240426 13:19:58.849232 27796 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:58.897729 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.849011 27496 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.897992 25029 rpcz_store.cc:276] 0426 13:19:56.108521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.115343 (+1006822us) service_pool.cc:225] Handling call
0426 13:19:57.115647 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:57.130996 (+ 15349us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:57.131042 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:57.172318 (+ 41276us) log.cc:842] Serialized 26 byte log entry
0426 13:19:58.435743 (+1263425us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:19:58.435780 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.435846 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:58.897244 (+461398us) raft_consensus.cc:1671] finished
0426 13:19:58.897263 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:58.897394 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25561,"raft.queue_time_us":46289,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":1229601,"threads_started":1,"wal-append.queue_time_us":1606684}
I20240426 13:19:58.886907 27850 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:58.878741 27862 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:19:58.897459 25069 connection.cc:232] RPC call timeout handler was delayed by -2.0278975679999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:19:58.901185 25030 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:58.857535 27778 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.878629 27677 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:58.886950 27867 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:58.878579 27883 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:19:58.817376 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.929307 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 112013us
W20240426 13:19:58.930222 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 145451us
I20240426 13:19:58.931010 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:58.908304 25069 connection.cc:232] RPC call timeout handler was delayed by -2.0278975679999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:19:58.876827 27699 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20240426 13:19:58.942222 25069 connection.cc:232] RPC call timeout handler was delayed by -1.9963975680000203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20240426 13:19:58.942569 25069 connection.cc:232] RPC call timeout handler was delayed by -1.88029756800006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20240426 13:19:58.915228 25104 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
W20240426 13:19:58.942831 25069 connection.cc:232] RPC call timeout handler was delayed by -1.7308975680000458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20240426 13:19:58.943147 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6982937919999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20240426 13:19:58.943429 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6921975679999832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20240426 13:19:58.943706 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6642975680000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20240426 13:19:58.944005 25069 connection.cc:232] RPC call timeout handler was delayed by -1.660697568000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20240426 13:19:58.944299 25069 connection.cc:232] RPC call timeout handler was delayed by -1.660697568000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20240426 13:19:58.944594 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6541937920000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
W20240426 13:19:58.944911 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6327975680000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20240426 13:19:58.945207 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6327975680000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20240426 13:19:58.945493 25069 connection.cc:232] RPC call timeout handler was delayed by -1.6210975679999819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20240426 13:19:58.945780 25069 connection.cc:232] RPC call timeout handler was delayed by -1.554497567999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20240426 13:19:58.946071 25069 connection.cc:232] RPC call timeout handler was delayed by -1.5391975679999632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20240426 13:19:58.946349 25069 connection.cc:232] RPC call timeout handler was delayed by -1.113497567999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25379999999999997s for a response
W20240426 13:19:58.946655 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0864975680000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25680000000000003s for a response
W20240426 13:19:58.946929 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0864975680000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25680000000000003s for a response
I20240426 13:19:58.947504 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.947405 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:58.936946 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:58.949195 25029 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:58.949740 25069 leader_election.cc:398] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:58.950042 25069 leader_election.cc:401] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:19:58.899757 27376 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:58.950632 27376 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.952241 26824 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7.
I20240426 13:19:58.952764 27376 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.953055 26824 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:58.954456 27706 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:19:58.954874 27706 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.955693 27706 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.958391 25624 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:19:58.959286 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:19:58.959959 25621 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:58.905213 25098 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:19:58.961871 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.962761 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:58.962960 24955 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:19:58.860810 27852 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:58.934306 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.961871 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.965152 24952 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:58.958555 27706 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:58.940594 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:58.983572 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.984771 25030 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:19:58.990412 27852 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:58.990713 27868 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:58.991636 24949 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:19:58.994741 26824 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:58.996310 27656 consensus_queue.cc:1046] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:19:58.997138 27656 consensus_queue.cc:1225] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:19:58.997730 27656 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:19:58.999212 27656 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:19:59.000375 27640 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.000833 27640 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:58.995824 26824 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.009927 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:58.926394 27677 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.009096 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:58.907343 27850 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.015605 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:58.925256 27867 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:59.019421 24949 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.019770 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:19:59.007295 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.021402 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.018456 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.002702 27676 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.001753 27640 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.037238 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.025084 25030 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:19:59.034924 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 94437.6us
I20240426 13:19:59.039819 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.041072 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 43677us
I20240426 13:19:59.041441 25029 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:19:59.043588 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.044014 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:19:59.016707 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 348267us
I20240426 13:19:59.057482 25030 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.057739 24955 raft_consensus.cc:2379] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:19:59.012588 26824 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.014906 27699 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240426 13:19:59.016726 25618 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 3 LEARNER]: Advancing to term 4
I20240426 13:19:59.067571 24949 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:59.067850 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.068639 24955 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:19:59.100771 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 62236.2us
W20240426 13:19:59.143276 25070 leader_election.cc:334] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.686s (SENT)
I20240426 13:19:59.148610 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.023875 27835 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:19:59.152302 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.161847 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.032683 27867 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:59.042318 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.002890 27881 consensus_queue.cc:1035] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.222s
W20240426 13:19:59.184332 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 82600.9us
I20240426 13:19:59.159062 27640 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.219832 27778 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.003572 27883 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.263746 27883 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.264343 24951 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20240426 13:19:59.265112 24951 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:19:59.265329 24951 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240426 13:19:59.249814 27889 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240426 13:19:59.251621 27878 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20240426 13:19:59.275708 27888 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:19:59.282038 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1475) took 2665 ms. Trace:
I20240426 13:19:59.282550 25103 rpcz_store.cc:276] 0426 13:19:56.616768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.616887 (+ 119us) service_pool.cc:225] Handling call
0426 13:19:56.635070 (+ 18183us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.115212 (+480142us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:57.115247 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:57.115308 (+ 61us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:57.115345 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:59.281747 (+2166402us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:19:59.281808 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:59.281959 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4285,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":1402,"spinlock_wait_cycles":1097023872}
I20240426 13:19:59.065174 27879 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.212s
I20240426 13:19:59.263901 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1596) took 3097 ms. Trace:
I20240426 13:19:59.265748 24951 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:19:59.226402 27835 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:19:59.268935 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.294606 24949 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.267367 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.291177 24954 rpcz_store.cc:276] 0426 13:19:56.166209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.166605 (+ 396us) service_pool.cc:225] Handling call
0426 13:19:59.263639 (+3097034us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b54003c37d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:59.263808 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73691904}
I20240426 13:19:59.224314 27474 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.296777 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:19:59.224694 27894 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:19:59.297654 25024 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:59.309917 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.311159 25024 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:19:59.150765 27740 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:19:59.295962 24948 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:19:59.300338 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.269742 27889 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.322657 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.285284 27350 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.327728 24949 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:19:59.334040 27894 consensus_queue.cc:1035] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240426 13:19:59.206758 25621 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:19:59.365200 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 309792us
I20240426 13:19:59.164592 27376 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.128s
I20240426 13:19:59.296365 27835 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.292333 27724 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.096s
I20240426 13:19:59.415891 24949 raft_consensus.cc:1261] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:19:59.392853 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 150286us
I20240426 13:19:59.393680 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:19:59.387475 27889 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.319974 27496 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:19:59.394762 27766 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.348480 27905 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20240426 13:19:59.391615 27796 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.428872 27796 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.429342 27796 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.373756 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.417124 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:19:59.395059 24954 raft_consensus.cc:2379] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:19:59.432075 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 58303.6us
I20240426 13:19:59.387516 27350 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:19:59.270629 26824 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.397171 27878 log.cc:924] Time spent T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240426 13:19:59.434851 24955 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:19:59.402776 27835 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:19:59.394343 27892 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:19:59.444977 27911 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240426 13:19:59.431407 27796 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.447786 25070 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.452605 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:19:59.454788 24948 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20240426 13:19:59.460114 25070 leader_election.cc:334] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671): Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 2.208s (SENT)
I20240426 13:19:59.460717 25070 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.446660 27638 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:19:59.466157 27350 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.472755 27630 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:19:59.473192 27630 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:59.473387 27350 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.469653 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1674) took 1643 ms. Trace:
I20240426 13:19:59.474757 25101 rpcz_store.cc:276] 0426 13:19:57.826533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.826686 (+ 153us) service_pool.cc:225] Handling call
0426 13:19:57.827101 (+ 415us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:19:58.299535 (+472434us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:58.299566 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:19:58.300121 (+ 555us) log.cc:842] Serialized 52 byte log entry
0426 13:19:58.875281 (+575160us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:19:58.941123 (+ 65842us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.941570 (+ 447us) log.cc:842] Serialized 14 byte log entry
0426 13:19:58.941788 (+ 218us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:58.941885 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:19:59.469364 (+527479us) raft_consensus.cc:1671] finished
0426 13:19:59.469394 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:19:59.469580 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":624657,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"raft.queue_time_us":128457,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":272,"thread_start_us":761514,"threads_started":3,"wal-append.queue_time_us":682498}
I20240426 13:19:59.461727 27707 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:59.486799 25024 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.496387 25070 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.496429 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.497495 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:19:59.464768 27881 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:19:59.465529 27638 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:19:59.501353 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:59.468163 27640 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:19:59.495306 24921 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.048s sys 0.084s
W20240426 13:19:59.502900 25100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.003s sys 0.098s
W20240426 13:19:59.503122 24921 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.048s sys 0.084s
W20240426 13:19:59.503279 25100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.003s sys 0.098s
W20240426 13:19:59.504985 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 574036us
I20240426 13:19:59.506275 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.507710 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.521677 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.522691 25024 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.480794 27327 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20240426 13:19:59.504662 27638 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:19:59.504878 27707 raft_consensus.cc:686] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.528542 27482 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.528004 27766 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:19:59.474078 27630 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.581725 27916 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:59.580549 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.551283 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.460968 27892 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.614714 27707 consensus_queue.cc:237] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.618327 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95185.1us
W20240426 13:19:59.618046 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 110428us
I20240426 13:19:59.623991 25024 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:19:59.584125 27881 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:19:59.629283 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.620939 27904 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20240426 13:19:59.640046 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.623558 27796 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:59.640425 24921 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.620596 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:19:59.641235 27796 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:19:59.641374 27640 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:19:59.628896 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1486) took 5583 ms. Trace:
I20240426 13:19:59.641846 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.627918 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.002s sys 0.023s
W20240426 13:19:59.646409 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.002s sys 0.023s
W20240426 13:19:59.647722 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 607975us
I20240426 13:19:59.647526 25022 rpcz_store.cc:276] 0426 13:19:54.041039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.800609 (+759570us) service_pool.cc:225] Handling call
0426 13:19:56.147917 (+1347308us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:56.171716 (+ 23799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:59.527112 (+3355396us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400055ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:19:59.624407 (+ 97295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961774976}
I20240426 13:19:59.649102 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.662249 27885 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
W20240426 13:19:59.662703 27908 consensus_queue.cc:1159] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
I20240426 13:19:59.691810 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:19:59.588047 27482 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.712162 27482 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:19:59.696370 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 47340.9us
I20240426 13:19:59.637161 27916 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.644896 27796 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:19:59.651154 27676 consensus_meta.cc:287] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.609s user 0.012s sys 0.000s
I20240426 13:19:59.647589 27640 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:19:59.704579 27855 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.826s user 0.009s sys 0.000s
W20240426 13:19:59.704707 27677 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.671s user 0.004s sys 0.004s
W20240426 13:19:59.707491 27850 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.671s user 0.001s sys 0.007s
W20240426 13:19:59.712363 27775 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.922s user 0.011s sys 0.000s
I20240426 13:19:59.722311 27850 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.723086 27775 raft_consensus.cc:2726] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:19:59.723532 25618 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.559s user 0.004s sys 0.004s
W20240426 13:19:59.708144 27860 log.cc:924] Time spent T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.060s user 0.000s sys 0.005s
I20240426 13:19:59.715584 25096 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 5. Ops: []
I20240426 13:19:59.702028 25024 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:19:59.716503 24995 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.719914 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:19:59.699154 27766 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:59.736048 25100 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:19:59.745234 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:19:59.745535 24995 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.745801 25096 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.746555 24995 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.737022 27850 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.733508 27677 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:19:59.721037 27676 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:19:59.737902 27916 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.739820 27835 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.740619 27714 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:19:59.742796 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.790009 27696 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.748220 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.825982 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 83259.9us
W20240426 13:19:59.790748 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44261.1us
I20240426 13:19:59.826900 24995 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.812618 25096 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:19:59.792773 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:19:59.837894 25024 raft_consensus.cc:2362] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9. Re-sending same reply.
I20240426 13:19:59.805121 27482 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:19:59.834442 27492 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:19:59.834789 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2031) took 1224 ms. Trace:
I20240426 13:19:59.835150 27722 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:19:59.748948 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:19:59.745725 25618 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20240426 13:19:59.792905 27775 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:19:59.740650 27907 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:19:59.720824 27640 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:19:59.746212 27766 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.874884 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:19:59.890640 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 250259us
I20240426 13:19:59.839766 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:19:59.814769 27924 log.cc:924] Time spent T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240426 13:19:59.826114 27714 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:19:59.872969 27492 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:19:59.926669 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86986.5us
W20240426 13:19:59.930773 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 55985us
I20240426 13:19:59.930864 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.724232 27855 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:19:59.907647 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:19:59.891863 27799 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.040s user 0.022s sys 0.053s
I20240426 13:19:59.934818 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.927577 24921 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.940448 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:19:59.927747 27766 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:19:59.941969 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:19:59.891570 25022 rpcz_store.cc:276] 0426 13:19:58.610439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.691575 (+1081136us) service_pool.cc:225] Handling call
0426 13:19:59.834727 (+143152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:19:59.876277 27850 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:19:59.893600 25618 pending_rounds.cc:76] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 3
I20240426 13:19:59.911579 27907 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:19:59.935189 27492 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:19:59.990561 25024 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:19:59.815033 27740 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:19:59.834478 27696 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:19:59.942584 27799 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.087s user 0.023s sys 0.054s
I20240426 13:19:59.936602 27498 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.042409 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:00.013224 27867 consensus_meta.cc:287] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.759s user 0.009s sys 0.000s
I20240426 13:19:59.923249 27775 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.009753 27862 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.082s user 0.005s sys 0.004s
W20240426 13:20:00.056555 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46918.4us
I20240426 13:20:00.058787 25022 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.039458 25618 pending_rounds.cc:97] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240426 13:20:00.042927 27907 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:00.043485 27631 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:00.009979 27746 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:19:59.835983 27881 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:00.054167 27898 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:00.055159 27850 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.058734 27912 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.069509 27498 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:00.077270 27900 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.087384 27898 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.100615 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.061919 27740 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.065001 25618 raft_consensus.cc:2866] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:00.103610 25022 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:00.105736 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.062816 27867 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.117127 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 182377us
I20240426 13:20:00.119630 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.101331 27631 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:00.107684 27898 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:00.124106 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.037s sys 0.087s
I20240426 13:20:00.099857 27892 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.123004 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.124897 25070 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:00.126123 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.037s sys 0.087s
I20240426 13:20:00.126700 25029 raft_consensus.cc:2379] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.099537 27928 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:00.115703 27677 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.132102 24995 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.067960 27862 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:00.121004 27900 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:00.135401 24995 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.078792 27714 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.136304 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:00.142488 25022 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.125348 27835 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:00.146186 25030 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:00.133474 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:00.141009 27900 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.150388 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.135567 27631 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:00.141700 27677 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.155529 24995 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.160282 24995 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.149837 27714 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.164287 27886 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:00.164749 27886 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:00.165133 27886 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.150490 27855 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:00.150871 27932 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240426 13:20:00.158679 27889 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.727s user 0.006s sys 0.004s
I20240426 13:20:00.149037 27498 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:00.175730 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 777942us
W20240426 13:20:00.187057 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 245149us
I20240426 13:20:00.156849 27631 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:00.162159 27835 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:00.154481 27900 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:00.152472 27482 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.187817 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:00.176517 27925 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.371s user 0.001s sys 0.001s
W20240426 13:20:00.180716 27906 log.cc:924] Time spent T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20240426 13:20:00.180310 27926 log.cc:924] Time spent T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
W20240426 13:20:00.171579 27921 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20240426 13:20:00.204892 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:00.166543 27696 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:00.178557 27350 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.704s user 0.009s sys 0.001s
I20240426 13:20:00.172410 27855 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:00.207115 27482 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.194293 27631 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:00.213258 24921 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.200680 27775 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.231267 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:00.242481 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 80670.6us
I20240426 13:20:00.243330 24995 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:00.243330 27722 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.163370 27740 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:00.245350 25022 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:00.246088 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2036) took 1568 ms. Trace:
I20240426 13:20:00.246549 25022 rpcz_store.cc:276] 0426 13:19:58.678106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.124362 (+1446256us) service_pool.cc:225] Handling call
0426 13:20:00.246017 (+121655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:00.242729 27942 raft_consensus.cc:2726] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.248587 25070 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.249195 27943 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:00.250206 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:00.251173 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.189594 27889 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.195369 27835 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:00.251296 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 63363.7us
I20240426 13:20:00.255214 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.220702 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:00.259737 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 510850us
W20240426 13:20:00.251224 27638 consensus_meta.cc:287] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.658s user 0.011s sys 0.000s
I20240426 13:20:00.243209 27940 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:00.154616 27850 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.210924 27936 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.269723 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.236428 27855 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.280774 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.281953 27935 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:00.250582 27903 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.202175 27932 consensus_queue.cc:1035] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240426 13:20:00.283044 27903 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:00.234485 27696 raft_consensus.cc:2741] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:00.271430 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.282652 27935 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.250825 27867 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.258086 27775 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:00.271667 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.275789 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.284924 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.287427 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:00.288252 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:00.288771 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:00.264909 27835 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:00.291005 25098 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.295789 25623 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:00.284372 24956 raft_consensus.cc:2362] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8. Re-sending same reply.
I20240426 13:20:00.293447 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.297101 24954 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:20:00.297315 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.299108 24949 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:00.301525 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:00.302253 24947 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:00.300453 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:00.310798 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62296.2us
I20240426 13:20:00.307145 24953 raft_consensus.cc:2362] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4. Re-sending same reply.
I20240426 13:20:00.291783 24996 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:00.338850 27677 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:00.290343 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:00.342209 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.288161 27903 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:00.346710 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.294548 27775 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:00.293464 27862 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:00.357944 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.275775 27638 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.373816 27933 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:00.313385 27867 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:00.374297 27855 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.312414 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:00.315507 27722 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.340567 27677 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:00.379747 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:00.452819 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102677us
I20240426 13:20:00.454923 25098 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.157054 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:01.083119 25096 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.792s user 0.009s sys 0.000s
I20240426 13:20:01.158962 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:00.693982 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 402228us
W20240426 13:20:00.694377 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 404088us
I20240426 13:20:01.152907 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:00.291914 27916 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:00.354048 27903 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:01.237838 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:01.242529 25096 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:01.242846 24955 raft_consensus.cc:2362] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4. Re-sending same reply.
I20240426 13:20:01.243669 24949 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:01.243280 24948 raft_consensus.cc:2379] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:01.151576 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:00.361780 27775 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:01.273295 24954 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:00.853013 27898 consensus_meta.cc:287] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.719s user 0.009s sys 0.000s
I20240426 13:20:01.366011 27898 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.849782 27640 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.917s user 0.004s sys 0.005s
I20240426 13:20:01.370682 27640 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.848063 27766 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.896s user 0.009s sys 0.001s
I20240426 13:20:01.374495 27766 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:00.854799 27796 consensus_meta.cc:287] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.130s user 0.008s sys 0.001s
I20240426 13:20:01.279166 27638 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:01.379645 27796 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
W20240426 13:20:01.391909 27714 consensus_meta.cc:287] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.223s user 0.010s sys 0.001s
W20240426 13:20:01.333931 24947 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.031s user 0.007s sys 0.003s
I20240426 13:20:01.426052 24947 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:01.426805 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1645) took 1139 ms (client timeout 1509 ms). Trace:
W20240426 13:20:01.427258 24947 rpcz_store.cc:269] 0426 13:20:00.286888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.295484 (+ 8596us) service_pool.cc:225] Handling call
0426 13:20:01.426732 (+1131248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:00.745041 27951 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20240426 13:20:01.465781 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.467687 24953 raft_consensus.cc:2396] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240426 13:20:00.711488 27740 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:00.711555 27862 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:00.921543 27946 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:00.459204 27889 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:00.625193 27828 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 3.060s user 0.003s sys 0.003s
I20240426 13:20:01.293951 27933 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:00.624965 27791 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 3.983s user 0.003s sys 0.010s
I20240426 13:20:01.237825 27867 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:01.502357 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 260116us
W20240426 13:20:01.321554 27886 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.156s user 0.005s sys 0.006s
W20240426 13:20:01.315511 27492 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.256s user 0.007s sys 0.002s
W20240426 13:20:01.322026 27835 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.009s user 0.010s sys 0.000s
W20240426 13:20:01.315447 27907 consensus_meta.cc:287] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.233s user 0.006s sys 0.004s
W20240426 13:20:01.324808 27631 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.088s user 0.009s sys 0.000s
I20240426 13:20:01.526332 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.562187 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:01.561996 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.569262 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:01.658295 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1961) took 1376 ms. Trace:
I20240426 13:20:01.659933 24947 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.589684 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1960) took 1307 ms. Trace:
I20240426 13:20:01.669533 24953 rpcz_store.cc:276] 0426 13:20:00.281838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.245014 (+963176us) service_pool.cc:225] Handling call
0426 13:20:01.589600 (+344586us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.415977 27916 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:01.673519 24948 raft_consensus.cc:2396] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240426 13:20:01.660593 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1643) took 1374 ms. Trace:
I20240426 13:20:01.684736 24947 rpcz_store.cc:276] 0426 13:20:00.286232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.525991 (+1239759us) service_pool.cc:225] Handling call
0426 13:20:01.660523 (+134532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.658843 24949 rpcz_store.cc:276] 0426 13:20:00.282060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.461564 (+1179504us) service_pool.cc:225] Handling call
0426 13:20:01.658220 (+196656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.660975 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1644) took 1374 ms. Trace:
I20240426 13:20:01.687211 24955 rpcz_store.cc:276] 0426 13:20:00.286574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.400373 (+1113799us) service_pool.cc:225] Handling call
0426 13:20:01.660907 (+260534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.690492 24954 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:01.690941 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1646) took 1401 ms. Trace:
I20240426 13:20:01.691187 24954 rpcz_store.cc:276] 0426 13:20:00.289610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.347948 (+ 58338us) service_pool.cc:225] Handling call
0426 13:20:01.690888 (+1342940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.693250 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 26) took 7223 ms. Trace:
I20240426 13:20:01.693769 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1962) took 1411 ms. Trace:
I20240426 13:20:01.964326 24948 rpcz_store.cc:276] 0426 13:20:00.282461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.400689 (+1118228us) service_pool.cc:225] Handling call
0426 13:20:01.693705 (+293016us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:01.966089 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.010s sys 0.018s
W20240426 13:20:01.966295 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 728522us
W20240426 13:20:01.641439 27900 consensus_meta.cc:287] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.430s user 0.015s sys 0.027s
W20240426 13:20:01.843132 25022 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.000s sys 0.032s
I20240426 13:20:01.967828 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:01.976801 24949 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:01.977699 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1965) took 1623 ms. Trace:
I20240426 13:20:01.977905 24995 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:01.978133 24949 rpcz_store.cc:276] 0426 13:20:00.354660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.694746 (+1340086us) service_pool.cc:225] Handling call
0426 13:20:01.969364 (+274618us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:01.977409 (+ 8045us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:01.977641 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:01.862890 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.008s sys 0.002s
W20240426 13:20:01.979796 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s user 0.008s sys 0.006s
W20240426 13:20:01.966614 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.010s sys 0.018s
W20240426 13:20:01.943967 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.015s sys 0.019s
I20240426 13:20:01.693758 25097 rpcz_store.cc:276] 0426 13:19:54.458202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.459851 (+ 1649us) service_pool.cc:225] Handling call
0426 13:20:01.681354 (+7221503us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b540001c2d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:01.681474 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5705892096}
W20240426 13:20:01.980818 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.016s sys 0.019s
W20240426 13:20:01.963579 27944 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.318s user 0.000s sys 0.001s
W20240426 13:20:01.984210 24994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.006s
W20240426 13:20:01.984694 24994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.000s sys 0.006s
W20240426 13:20:01.953045 27954 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20240426 13:20:01.624814 27498 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.873s user 0.001s sys 0.011s
I20240426 13:20:01.576426 27862 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:01.976866 25022 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.697s user 0.001s sys 0.034s
I20240426 13:20:01.807202 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 532) took 1414 ms. Trace:
I20240426 13:20:01.997927 24955 rpcz_store.cc:276] 0426 13:20:00.392622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.687761 (+1295139us) service_pool.cc:225] Handling call
0426 13:20:01.806536 (+118775us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:01.806775 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:01.806812 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:01.806868 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:01.806917 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:01.806982 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:01.807124 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.561935 27939 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.567806 27740 raft_consensus.cc:686] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:01.992475 27957 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:01.671594 27913 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:01.888820 27947 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.195s user 0.000s sys 0.000s
I20240426 13:20:01.578720 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 579) took 1285 ms. Trace:
I20240426 13:20:01.574151 27933 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.006511 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 378) took 4819 ms. Trace:
I20240426 13:20:01.692927 26824 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:01.746657 27303 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.591395 27945 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:02.020514 25589 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.674s (SENT) [suppressed 29 similar messages]
I20240426 13:20:01.685978 27492 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:02.002322 27498 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.263s user 0.003s sys 0.014s
W20240426 13:20:02.047060 27948 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20240426 13:20:01.987761 27898 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.030454 27867 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.025503 27960 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:01.837852 27950 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:02.030239 25617 rpcz_store.cc:276] 0426 13:20:00.281090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.284576 (+ 3486us) service_pool.cc:225] Handling call
0426 13:20:01.566095 (+1281519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:01.676788 27631 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.671504 27886 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.009843 27850 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.039353 27686 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:02.039409 27303 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.687515 27835 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.931010 27935 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:02.044764 27820 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 4.979s user 0.025s sys 0.043s
I20240426 13:20:01.990379 27900 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.017954 27916 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.012925 27796 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.030370 27913 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.011003 27862 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.843211 27962 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.044209 27640 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.059007 27740 consensus_queue.cc:237] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:01.676508 27907 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.062017 27686 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.072364 27303 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:01.718205 27903 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:02.049316 27945 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.056864 27898 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.016144 27939 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.058028 27950 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.034955 25022 rpcz_store.cc:276] 0426 13:19:57.187602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.276316 (+3088714us) service_pool.cc:225] Handling call
0426 13:20:00.296402 (+ 20086us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:00.309528 (+ 13126us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:00.309573 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:00.320909 (+ 11336us) log.cc:842] Serialized 26 byte log entry
0426 13:20:01.154020 (+833111us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:01.997538 (+843518us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.006038 (+ 8500us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.006146 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.006209 (+ 63us) raft_consensus.cc:1671] finished
0426 13:20:02.006228 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.006418 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20716,"raft.queue_time_us":392942,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":250530,"thread_start_us":718705,"threads_started":2,"wal-append.queue_time_us":1365146}
I20240426 13:20:02.073969 27835 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.324378 27710 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.275538 25025 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1489) took 8198 ms (client timeout 1661 ms). Trace:
W20240426 13:20:02.216799 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1482) took 8202 ms (client timeout 1527 ms). Trace:
W20240426 13:20:02.274582 25029 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2029) took 3664 ms (client timeout 2065 ms). Trace:
I20240426 13:20:02.269470 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2012) took 6775 ms. Trace:
I20240426 13:20:02.227897 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 536) took 1074 ms. Trace:
W20240426 13:20:02.271189 25027 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1511) took 4261 ms (client timeout 2040 ms). Trace:
I20240426 13:20:02.226070 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 534) took 1779 ms. Trace:
W20240426 13:20:02.273290 25024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2028) took 3663 ms (client timeout 2832 ms). Trace:
I20240426 13:20:02.206857 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1593) took 6535 ms. Trace:
I20240426 13:20:02.344895 24948 rpcz_store.cc:276] 0426 13:20:00.446932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.980437 (+1533505us) service_pool.cc:225] Handling call
0426 13:20:02.224890 (+244453us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.225454 (+ 564us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.225509 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.225573 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:02.225629 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.225719 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.225879 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.342797 25023 rpcz_store.cc:276] 0426 13:19:55.493590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.736541 (+1242951us) service_pool.cc:225] Handling call
0426 13:19:56.736894 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:57.174225 (+437331us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.268824 (+5094599us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540025f654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.269256 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":993131776}
I20240426 13:20:02.326491 27916 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.342963 24955 rpcz_store.cc:276] 0426 13:20:01.152878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.999398 (+846520us) service_pool.cc:225] Handling call
0426 13:20:02.226885 (+227487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.227326 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.227372 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.227448 (+ 76us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:02.227500 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.227571 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.227701 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:02.345069 25024 rpcz_store.cc:269] 0426 13:19:58.609633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.839743 (+1230110us) service_pool.cc:225] Handling call
0426 13:20:02.272762 (+2433019us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b54003f7354. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.273059 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122441472}
W20240426 13:20:02.342295 25031 rpcz_store.cc:269] 0426 13:19:54.013909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.026563 (+ 12654us) service_pool.cc:225] Handling call
0426 13:20:02.215778 (+8189215us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540032c8d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.216504 (+ 726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4754335616}
W20240426 13:20:02.343194 25027 rpcz_store.cc:269] 0426 13:19:58.009803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.025261 (+ 15458us) service_pool.cc:225] Handling call
0426 13:20:02.270769 (+4245508us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54003ae3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.271019 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":277254400}
W20240426 13:20:02.342446 25029 rpcz_store.cc:269] 0426 13:19:58.609922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.161298 (+551376us) service_pool.cc:225] Handling call
0426 13:20:02.274104 (+3112806us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x7b540025fb54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.274377 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2215829888}
I20240426 13:20:02.288398 27708 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.342137 25025 rpcz_store.cc:269] 0426 13:19:54.077519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:54.208860 (+131341us) service_pool.cc:225] Handling call
0426 13:20:02.275101 (+8066241us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.275353 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5057163648}
W20240426 13:20:02.361647 25023 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 390) took 2073 ms (client timeout 1653 ms). Trace:
I20240426 13:20:02.371979 25027 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:02.067894 27886 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.372731 25023 rpcz_store.cc:269] 0426 13:20:00.287882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.361245 (+2073363us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:02.361542 (+ 297us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:02.068464 27631 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.379739 25028 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:02.371487 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.353302 27710 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.385082 27907 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.385165 25029 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:02.077198 27492 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.322953 27933 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.390494 25101 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:02.266455 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:02.365312 25587 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.010s sys 0.007s
I20240426 13:20:02.399571 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1495) took 4589 ms. Trace:
I20240426 13:20:02.400151 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1497) took 4589 ms. Trace:
I20240426 13:20:02.403370 25029 rpcz_store.cc:276] 0426 13:19:57.801943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.384241 (+4582298us) service_pool.cc:225] Handling call
0426 13:20:02.384866 (+ 625us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.386462 (+ 1596us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.391639 (+ 5177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.402614 25027 rpcz_store.cc:276] 0426 13:19:57.801391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.370790 (+4569399us) service_pool.cc:225] Handling call
0426 13:20:02.371610 (+ 820us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.388811 (+ 17201us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.391132 (+ 2321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71296}
I20240426 13:20:02.399035 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:02.403064 25587 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.802s user 0.011s sys 0.007s
I20240426 13:20:02.063642 27960 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.407878 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:02.402969 25071 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.000s sys 0.060s
I20240426 13:20:02.391214 27710 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.412199 25097 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:02.413674 27933 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.416682 25071 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.061s
W20240426 13:20:02.417716 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.011s sys 0.042s
I20240426 13:20:02.421113 24949 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:02.422346 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s user 0.011s sys 0.043s
I20240426 13:20:02.421686 25096 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:02.425397 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:02.426993 24953 raft_consensus.cc:2379] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:02.427189 27903 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:02.428416 26996 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:02.429028 26996 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:02.429226 27913 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.427191 27945 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.430755 26996 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.431571 27683 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.432569 27683 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.435390 27683 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.442713 27916 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.445574 27916 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.448004 27900 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.448429 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.448944 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:02.423266 27946 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.010s sys 0.026s
I20240426 13:20:02.346225 24950 rpcz_store.cc:276] 0426 13:19:55.670699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:55.670817 (+ 118us) service_pool.cc:225] Handling call
0426 13:19:55.671080 (+ 263us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:55.728336 (+ 57256us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:55.728382 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:19:55.775242 (+ 46860us) log.cc:842] Serialized 26 byte log entry
0426 13:19:56.400043 (+624801us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:19:56.400081 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:19:56.400148 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:00.294445 (+3894297us) spinlock_profiling.cc:243] Waited 8.84 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:02.205870 (+1911425us) raft_consensus.cc:1671] finished
0426 13:20:02.205949 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.206487 (+ 538us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56865,"raft.queue_time_us":63068,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":121419,"spinlock_wait_cycles":20340608,"thread_start_us":243076,"threads_started":1,"wal-append.queue_time_us":446915}
I20240426 13:20:02.464831 25617 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 4: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:02.469115 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.461627 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1963) took 2169 ms. Trace:
W20240426 13:20:02.471001 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 45497.3us
I20240426 13:20:02.471850 24953 rpcz_store.cc:276] 0426 13:20:00.282749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.688227 (+1405478us) service_pool.cc:225] Handling call
0426 13:20:02.265947 (+577720us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:02.451705 (+185758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3938176}
I20240426 13:20:02.472831 27913 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240426 13:20:02.477326 27913 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:02.477783 27913 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.464926 27960 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.457376 27946 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.483397 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:02.479638 27472 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.004s
I20240426 13:20:02.492206 27736 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:02.492748 27736 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:02.481904 27976 log.cc:924] Time spent T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20240426 13:20:02.503927 26996 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:02.504688 24950 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:02.453639 27708 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.505558 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1968) took 1344 ms. Trace:
I20240426 13:20:02.505981 24950 rpcz_store.cc:276] 0426 13:20:01.161129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.473422 (+1312293us) service_pool.cc:225] Handling call
0426 13:20:02.500354 (+ 26932us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.505183 (+ 4829us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.505463 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.508052 24950 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:02.508292 27886 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.508734 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1969) took 1347 ms. Trace:
I20240426 13:20:02.508868 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 508) took 3721 ms. Trace:
I20240426 13:20:02.509178 27886 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.509930 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.510277 24953 raft_consensus.cc:1226] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:02.510573 27710 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.504932 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:02.479342 27959 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20240426 13:20:02.511471 27710 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.512653 27886 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.511013 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1970) took 1349 ms. Trace:
I20240426 13:20:02.509370 25102 rpcz_store.cc:276] 0426 13:19:58.787589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.824165 (+ 36576us) service_pool.cc:225] Handling call
0426 13:19:58.824538 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.868205 (+ 43667us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:19:58.868236 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:58.868279 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:19:58.868309 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.508350 (+3640041us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b54003ee9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.508411 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.508680 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99872000}
I20240426 13:20:02.513511 24953 rpcz_store.cc:276] 0426 13:20:01.161698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.509522 (+1347824us) service_pool.cc:225] Handling call
0426 13:20:02.510008 (+ 486us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.510748 (+ 740us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.510940 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.515794 27970 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.509207 24950 rpcz_store.cc:276] 0426 13:20:01.161405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.507154 (+1345749us) service_pool.cc:225] Handling call
0426 13:20:02.507723 (+ 569us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.508442 (+ 719us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.508652 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.516518 27970 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.509284 27492 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.522089 27492 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.523286 27640 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:02.525188 27492 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.525310 27710 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:02.525826 27775 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.108s user 0.015s sys 0.004s
I20240426 13:20:02.526420 27775 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.527350 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 574) took 3570 ms. Trace:
I20240426 13:20:02.527941 25618 rpcz_store.cc:276] 0426 13:19:58.956727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.956850 (+ 123us) service_pool.cc:225] Handling call
0426 13:19:58.957178 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:00.120155 (+1162977us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.526721 (+2406566us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b54007503d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.527122 (+ 401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111197568}
I20240426 13:20:02.535566 27970 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.536757 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.535468 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.530078 24947 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:02.539793 27844 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:20:02.539942 25624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 417) took 2254 ms (client timeout 1683 ms). Trace:
I20240426 13:20:02.502094 27898 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.543140 24948 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:02.547377 25624 rpcz_store.cc:269] 0426 13:20:00.285542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.285672 (+ 130us) service_pool.cc:225] Handling call
0426 13:20:02.535071 (+2249399us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:02.539844 (+ 4773us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6336256}
W20240426 13:20:02.550755 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49919.3us
I20240426 13:20:02.541399 27913 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.568609 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1966) took 1407 ms. Trace:
I20240426 13:20:02.556208 27862 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.565701 27898 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.569953 24948 rpcz_store.cc:276] 0426 13:20:01.160513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.424668 (+1264155us) service_pool.cc:225] Handling call
0426 13:20:02.478901 (+ 54233us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.566162 (+ 87261us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.568218 (+ 2056us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.567886 27886 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20240426 13:20:02.585346 27775 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.585086 27862 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:02.597824 27941 log.cc:924] Time spent T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 2.236s user 0.003s sys 0.031s
I20240426 13:20:02.594482 27913 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:02.600209 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 90470.8us
I20240426 13:20:02.602699 27487 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:02.588189 27950 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.609059 24955 raft_consensus.cc:1261] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:02.609666 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1967) took 1448 ms. Trace:
I20240426 13:20:02.588495 27482 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.611192 24955 rpcz_store.cc:276] 0426 13:20:01.160840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.608165 (+1447325us) service_pool.cc:225] Handling call
0426 13:20:02.608616 (+ 451us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.609420 (+ 804us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.609619 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.614970 25022 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:02.615716 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2025) took 4007 ms. Trace:
I20240426 13:20:02.620213 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 535) took 1468 ms. Trace:
I20240426 13:20:02.620570 24949 rpcz_store.cc:276] 0426 13:20:01.152188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.993031 (+840843us) service_pool.cc:225] Handling call
0426 13:20:01.993455 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.440861 (+447406us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.619861 (+179000us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54002cabd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.620145 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":969883008}
I20240426 13:20:02.615978 25022 rpcz_store.cc:276] 0426 13:19:58.608787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.614095 (+4005308us) service_pool.cc:225] Handling call
0426 13:20:02.614569 (+ 474us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.615456 (+ 887us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.615658 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:02.622198 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.623118 25022 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:02.624428 27656 consensus_queue.cc:1046] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:02.626143 27513 raft_consensus.cc:3032] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:02.626618 27513 raft_consensus.cc:729] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:02.627786 27513 consensus_queue.cc:260] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:02.628886 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 384) took 3854 ms (client timeout 1924 ms). Trace:
I20240426 13:20:02.628939 27513 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:02.629530 25030 rpcz_store.cc:269] 0426 13:19:58.774710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.299817 (+525107us) service_pool.cc:225] Handling call
0426 13:20:02.628486 (+3328669us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b5400242f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.628699 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1907150208}
I20240426 13:20:02.630671 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.632548 27715 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:02.633083 27715 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:02.634603 27975 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.205s user 0.001s sys 0.001s
I20240426 13:20:02.623975 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.637128 25022 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
W20240426 13:20:02.640935 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139280us
I20240426 13:20:02.643112 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.642062 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.644762 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:02.608729 27946 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:02.649122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 46841.1us
I20240426 13:20:02.649755 27855 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.649998 27722 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:02.617580 27939 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.655484 27862 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.656430 27686 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:02.656955 27482 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.684258 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.685575 27482 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.677102 25030 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:02.686093 27722 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.687201 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:02.688325 27656 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:02.688655 27722 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.688872 27656 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:02.689517 27656 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:02.677500 25022 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:02.692567 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:02.698688 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.699420 27687 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:02.703336 27724 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:02.702661 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.704016 27950 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:02.704720 27950 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:02.704792 25624 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:02.705143 27950 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.705950 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.709758 27868 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:02.709951 25096 raft_consensus.cc:1261] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:02.710287 27868 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:02.708640 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.710428 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.726805 25026 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:02.731690 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:02.735834 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 93576.6us
I20240426 13:20:02.716411 27715 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.743781 27715 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.748512 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 522) took 2849 ms. Trace:
I20240426 13:20:02.735224 27487 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:02.749222 27487 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:02.749814 27487 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:02.748927 24952 rpcz_store.cc:276] 0426 13:19:59.899096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.899237 (+ 141us) service_pool.cc:225] Handling call
0426 13:19:59.899631 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:00.218798 (+319167us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:00.218826 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:00.221326 (+ 2500us) log.cc:842] Serialized 26 byte log entry
0426 13:20:00.311801 (+ 90475us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:00.311841 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.747925 (+2436084us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540004ae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.747988 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.748058 (+ 70us) raft_consensus.cc:1671] finished
0426 13:20:02.748072 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.748330 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14876,"raft.queue_time_us":86888,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":20534,"spinlock_wait_cycles":404918016,"thread_start_us":232846,"threads_started":2,"wal-append.queue_time_us":1354832}
I20240426 13:20:02.748179 27656 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.753832 27862 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.753772 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.756156 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 533) took 2328 ms. Trace:
I20240426 13:20:02.757611 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.753649 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:02.757688 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.762636 25097 raft_consensus.cc:595] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.776189 25097 raft_consensus.cc:603] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:02.764672 27833 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:02.755807 27950 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:02.779732 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:02.775317 24954 rpcz_store.cc:276] 0426 13:20:00.427321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.691632 (+1264311us) service_pool.cc:225] Handling call
0426 13:20:02.755027 (+1063395us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:02.755348 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.755691 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.755719 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:02.755758 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:02.755786 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.755839 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.755959 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2396544}
I20240426 13:20:02.780547 25030 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.711977 27855 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.782027 25022 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:02.792312 25105 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1488) took 5657 ms (client timeout 1886 ms). Trace:
I20240426 13:20:02.792574 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:02.792934 25105 rpcz_store.cc:269] 0426 13:19:57.134210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:57.135173 (+ 963us) service_pool.cc:225] Handling call
0426 13:20:02.791718 (+5656545us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54002d2654. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:02.792009 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1536800640}
I20240426 13:20:02.794314 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:02.795020 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1971) took 1633 ms. Trace:
I20240426 13:20:02.795439 24953 rpcz_store.cc:276] 0426 13:20:01.161963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.528133 (+1366170us) service_pool.cc:225] Handling call
0426 13:20:02.559544 (+ 31411us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.560159 (+ 615us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:02.560196 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:02.560944 (+ 748us) log.cc:842] Serialized 26 byte log entry
0426 13:20:02.697832 (+136888us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:02.697876 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.697939 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:02.794741 (+ 96802us) raft_consensus.cc:1671] finished
0426 13:20:02.794774 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:02.794940 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":136436,"threads_started":1,"wal-append.queue_time_us":152082}
I20240426 13:20:02.796790 27766 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:02.795061 25096 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:02.783205 27833 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:02.799134 27487 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:02.800704 25624 raft_consensus.cc:2396] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:02.787876 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:02.822325 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:02.822563 27487 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:02.823499 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.823727 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:02.829244 27487 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:02.785264 27768 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:02.836884 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 49813.1us
I20240426 13:20:02.848563 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1964) took 2565 ms. Trace:
I20240426 13:20:02.849107 24947 rpcz_store.cc:276] 0426 13:20:00.283036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:01.694986 (+1411950us) service_pool.cc:225] Handling call
0426 13:20:01.731220 (+ 36234us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.553235 (+822015us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.832268 (+279033us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540038f9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:02.848485 (+ 16217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1832775936}
I20240426 13:20:02.861980 25096 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:03.393020 27923 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:03.186132 27868 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:03.393991 27923 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:04.096516 27980 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.575s user 0.001s sys 0.000s
I20240426 13:20:04.080456 27855 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:04.146396 27981 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.358s user 0.002s sys 0.000s
W20240426 13:20:04.418555 27981 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20240426 13:20:04.458138 27868 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.473433 27950 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:04.484431 27984 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.664s user 0.000s sys 0.002s
I20240426 13:20:04.525707 25031 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.526314 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1499) took 6720 ms. Trace:
I20240426 13:20:04.526540 25031 rpcz_store.cc:276] 0426 13:19:57.806206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.525197 (+6718991us) service_pool.cc:225] Handling call
0426 13:20:04.525554 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.526107 (+ 553us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.526254 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:04.527370 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2073) took 1675 ms (client timeout 1660 ms). Trace:
W20240426 13:20:04.527644 25031 rpcz_store.cc:269] 0426 13:20:02.851985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.527227 (+1675242us) service_pool.cc:212] Skipping call since client already timed out
0426 13:20:04.527317 (+ 90us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240426 13:20:04.528510 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.529534 25031 raft_consensus.cc:2396] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20240426 13:20:04.533878 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 395) took 2020 ms (client timeout 2031 ms). Trace:
W20240426 13:20:04.534142 25031 rpcz_store.cc:269] 0426 13:20:02.513101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.528238 (+2015137us) service_pool.cc:225] Handling call
0426 13:20:04.533806 (+ 5568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.535084 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:04.535876 25031 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.540807 25031 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:04.541168 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2056) took 2010 ms (client timeout 2090 ms). Trace:
W20240426 13:20:04.542984 25031 rpcz_store.cc:269] 0426 13:20:02.530971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.534844 (+2003873us) service_pool.cc:225] Handling call
0426 13:20:04.541117 (+ 6273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.544806 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:04.545580 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2068) took 1897 ms (client timeout 1993 ms). Trace:
W20240426 13:20:04.546561 25031 rpcz_store.cc:269] 0426 13:20:02.648664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.544534 (+1895870us) service_pool.cc:225] Handling call
0426 13:20:04.545530 (+ 996us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.547470 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.548082 25031 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:04.548545 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47174 (request call id 396) took 1927 ms (client timeout 2035 ms). Trace:
W20240426 13:20:04.548885 25031 rpcz_store.cc:269] 0426 13:20:02.621325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.547181 (+1925856us) service_pool.cc:225] Handling call
0426 13:20:04.548498 (+ 1317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.549814 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:04.550546 25031 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:04.559969 25031 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:20:04.561026 25031 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2058) took 2029 ms (client timeout 2193 ms). Trace:
W20240426 13:20:04.561465 25031 rpcz_store.cc:269] 0426 13:20:02.531624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.549542 (+2017918us) service_pool.cc:225] Handling call
0426 13:20:04.560967 (+ 11425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.563057 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:04.563809 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2070) took 1836 ms. Trace:
I20240426 13:20:04.564201 25031 rpcz_store.cc:276] 0426 13:20:02.727478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.562738 (+1835260us) service_pool.cc:225] Handling call
0426 13:20:04.563760 (+ 1022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.565521 25031 raft_consensus.cc:1226] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.566161 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2035) took 5892 ms. Trace:
I20240426 13:20:04.566542 25031 rpcz_store.cc:276] 0426 13:19:58.673511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.564827 (+5891316us) service_pool.cc:225] Handling call
0426 13:20:04.565232 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.565830 (+ 598us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.566111 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.568146 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 380) took 5891 ms. Trace:
I20240426 13:20:04.568465 25031 rpcz_store.cc:276] 0426 13:19:58.676798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.567217 (+5890419us) service_pool.cc:225] Handling call
0426 13:20:04.567588 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.567805 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.567836 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.567881 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:04.567911 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.567961 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.568082 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.570022 25031 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20240426 13:20:04.570700 25031 pending_rounds.cc:76] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 6
I20240426 13:20:04.570952 25031 pending_rounds.cc:97] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240426 13:20:04.571192 25031 raft_consensus.cc:2866] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.571817 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 381) took 5894 ms. Trace:
I20240426 13:20:04.572217 25031 rpcz_store.cc:276] 0426 13:19:58.677095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.569279 (+5892184us) service_pool.cc:225] Handling call
0426 13:20:04.569621 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.571611 (+ 1990us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.571762 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.582791 27928 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:04.635057 27962 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.635814 27962 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.638080 27962 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:04.639407 27716 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.640004 27716 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.642200 27716 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.638609 27962 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.644140 27962 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.646884 27962 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:04.663463 25030 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.882s user 0.011s sys 0.018s
I20240426 13:20:04.664605 25030 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:04.665345 25030 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2059) took 2133 ms (client timeout 1941 ms). Trace:
I20240426 13:20:04.666373 25031 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:04.666299 25030 rpcz_store.cc:269] 0426 13:20:02.531874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.779465 (+247591us) service_pool.cc:225] Handling call
0426 13:20:04.665248 (+1885783us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.668303 27656 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.668519 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 385) took 5893 ms. Trace:
I20240426 13:20:04.668846 25030 rpcz_store.cc:276] 0426 13:19:58.776002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.667599 (+5891597us) service_pool.cc:225] Handling call
0426 13:20:04.667942 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.668196 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.668219 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.668272 (+ 53us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:04.668302 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.668354 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.668469 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.670658 25030 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 10. Ops: []
I20240426 13:20:04.671293 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 386) took 5742 ms. Trace:
I20240426 13:20:04.671631 25030 rpcz_store.cc:276] 0426 13:19:58.928987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.670164 (+5741177us) service_pool.cc:225] Handling call
0426 13:20:04.670485 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.671070 (+ 585us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.671244 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.698518 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2037) took 5304 ms. Trace:
I20240426 13:20:04.699075 25030 rpcz_store.cc:276] 0426 13:19:59.394257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.673090 (+5278833us) service_pool.cc:225] Handling call
0426 13:20:04.673432 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.674103 (+ 671us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.674132 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.674609 (+ 477us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.697974 (+ 23365us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:04.698020 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.698292 (+ 272us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.698340 (+ 48us) raft_consensus.cc:1671] finished
0426 13:20:04.698355 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.698463 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":22975,"threads_started":1,"wal-append.queue_time_us":20469}
I20240426 13:20:04.701490 25030 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:04.703655 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2038) took 5309 ms. Trace:
I20240426 13:20:04.704107 25030 rpcz_store.cc:276] 0426 13:19:59.394551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.700752 (+5306201us) service_pool.cc:225] Handling call
0426 13:20:04.701196 (+ 444us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.701922 (+ 726us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.703588 (+ 1666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.705611 25030 raft_consensus.cc:1226] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240426 13:20:04.706293 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2039) took 5311 ms. Trace:
I20240426 13:20:04.706648 25030 rpcz_store.cc:276] 0426 13:19:59.394841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.704980 (+5310139us) service_pool.cc:225] Handling call
0426 13:20:04.705381 (+ 401us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.706059 (+ 678us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.706241 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.710042 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1507) took 6729 ms. Trace:
I20240426 13:20:04.710507 25025 rpcz_store.cc:276] 0426 13:19:57.980553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.708853 (+6728300us) service_pool.cc:225] Handling call
0426 13:20:04.709181 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.709608 (+ 427us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.709650 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.709708 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.709753 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.709845 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.709980 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.712340 25025 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:04.713009 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2041) took 5317 ms. Trace:
I20240426 13:20:04.713372 25025 rpcz_store.cc:276] 0426 13:19:59.395419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.711681 (+5316262us) service_pool.cc:225] Handling call
0426 13:20:04.712109 (+ 428us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.712752 (+ 643us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.712950 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.715032 25025 raft_consensus.cc:1226] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.717656 25029 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:04.718492 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1509) took 6719 ms. Trace:
I20240426 13:20:04.718881 25029 rpcz_store.cc:276] 0426 13:19:57.999507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.716968 (+6717461us) service_pool.cc:225] Handling call
0426 13:20:04.717423 (+ 455us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.718210 (+ 787us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.718426 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:04.724769 25099 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55910 (request call id 519) took 2338 ms (client timeout 1653 ms). Trace:
W20240426 13:20:04.725382 25099 rpcz_store.cc:269] 0426 13:20:02.385870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.406102 (+ 20232us) service_pool.cc:225] Handling call
0426 13:20:04.724419 (+2318317us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b540016a154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.724649 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":957015936}
I20240426 13:20:04.715776 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2042) took 5320 ms. Trace:
I20240426 13:20:04.729290 25025 rpcz_store.cc:276] 0426 13:19:59.395680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.714198 (+5318518us) service_pool.cc:225] Handling call
0426 13:20:04.714669 (+ 471us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.715484 (+ 815us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.715719 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.730902 25025 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:04.731763 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2045) took 5335 ms. Trace:
I20240426 13:20:04.732270 25025 rpcz_store.cc:276] 0426 13:19:59.396522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.730191 (+5333669us) service_pool.cc:225] Handling call
0426 13:20:04.730625 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.731424 (+ 799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.731622 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.733728 25025 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:04.734378 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 387) took 5259 ms. Trace:
I20240426 13:20:04.734766 25025 rpcz_store.cc:276] 0426 13:19:59.475426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.733029 (+5257603us) service_pool.cc:225] Handling call
0426 13:20:04.733362 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.734099 (+ 737us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.734317 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.738579 25025 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:04.738720 27911 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.739192 27911 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.739303 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2050) took 5253 ms. Trace:
I20240426 13:20:04.739708 25025 rpcz_store.cc:276] 0426 13:19:59.486198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.737954 (+5251756us) service_pool.cc:225] Handling call
0426 13:20:04.738320 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.739075 (+ 755us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.739257 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.740473 27911 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.741469 25025 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.741983 27911 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.742184 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2051) took 5255 ms. Trace:
I20240426 13:20:04.742923 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:04.742682 25025 rpcz_store.cc:276] 0426 13:19:59.486463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.740664 (+5254201us) service_pool.cc:225] Handling call
0426 13:20:04.741034 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.741946 (+ 912us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.742118 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.743592 25623 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:04.756505 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2040) took 5361 ms. Trace:
I20240426 13:20:04.757104 25030 rpcz_store.cc:276] 0426 13:19:59.395144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.707537 (+5312393us) service_pool.cc:225] Handling call
0426 13:20:04.707917 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.708618 (+ 701us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.708645 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.709121 (+ 476us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.755763 (+ 46642us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:04.755834 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.755910 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.755966 (+ 56us) raft_consensus.cc:1671] finished
0426 13:20:04.755978 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.756422 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":46201,"threads_started":1,"wal-append.queue_time_us":35451}
I20240426 13:20:04.759290 25030 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:04.760447 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2053) took 5122 ms. Trace:
I20240426 13:20:04.760864 25030 rpcz_store.cc:276] 0426 13:19:59.638527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.758691 (+5120164us) service_pool.cc:225] Handling call
0426 13:20:04.759104 (+ 413us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.760224 (+ 1120us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.760392 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.763392 25623 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:04.764124 25623 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44520 (request call id 427) took 1925 ms. Trace:
I20240426 13:20:04.764447 25623 rpcz_store.cc:276] 0426 13:20:02.838558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.844428 (+ 5870us) service_pool.cc:225] Handling call
0426 13:20:04.742648 (+1898220us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:04.764011 (+ 21363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14691200}
I20240426 13:20:04.766175 27708 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.783094 27725 raft_consensus.cc:465] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Not starting pre-election -- already a leader
I20240426 13:20:04.783679 27725 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.784194 27725 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.787004 27725 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.793334 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 389) took 5288 ms. Trace:
W20240426 13:20:04.794826 24995 connection.cc:232] RPC call timeout handler was delayed by -1.8494979160000184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2959s for a response
I20240426 13:20:04.793946 25025 rpcz_store.cc:276] 0426 13:19:59.505358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.744152 (+5238794us) service_pool.cc:225] Handling call
0426 13:20:04.746772 (+ 2620us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:04.747552 (+ 780us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.747583 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.748076 (+ 493us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.770793 (+ 22717us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.792167 (+ 21374us) log.cc:842] Serialized 14 byte log entry
0426 13:20:04.792953 (+ 786us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.793078 (+ 125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.793127 (+ 49us) raft_consensus.cc:1671] finished
0426 13:20:04.793141 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.793276 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":517,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":43266,"threads_started":2,"wal-append.queue_time_us":22187}
W20240426 13:20:04.796396 24995 connection.cc:232] RPC call timeout handler was delayed by -1.3009583929999735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20240426 13:20:04.798056 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 392) took 2585 ms. Trace:
I20240426 13:20:04.798434 25025 rpcz_store.cc:276] 0426 13:20:02.212939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.797157 (+2584218us) service_pool.cc:225] Handling call
0426 13:20:04.797507 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.797735 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.797763 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.797808 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:04.797845 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.797900 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.798005 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.803896 25025 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:04.804553 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 394) took 2291 ms. Trace:
I20240426 13:20:04.804904 25025 rpcz_store.cc:276] 0426 13:20:02.512745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.803184 (+2290439us) service_pool.cc:225] Handling call
0426 13:20:04.803522 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.804311 (+ 789us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.804487 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.806794 25025 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:20:04.807415 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2057) took 2276 ms. Trace:
I20240426 13:20:04.807788 25025 rpcz_store.cc:276] 0426 13:20:02.531333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.805982 (+2274649us) service_pool.cc:225] Handling call
0426 13:20:04.806371 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.807164 (+ 793us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.807361 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.809327 25025 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:04.809896 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2069) took 2105 ms. Trace:
I20240426 13:20:04.810248 25025 rpcz_store.cc:276] 0426 13:20:02.704433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.808599 (+2104166us) service_pool.cc:225] Handling call
0426 13:20:04.808929 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.809676 (+ 747us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.809845 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.811740 25025 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.812402 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 397) took 2066 ms. Trace:
I20240426 13:20:04.812726 25025 rpcz_store.cc:276] 0426 13:20:02.745852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.810993 (+2065141us) service_pool.cc:225] Handling call
0426 13:20:04.811469 (+ 476us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.812206 (+ 737us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.812356 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.814450 25025 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:04.815080 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2071) took 2026 ms. Trace:
I20240426 13:20:04.815419 25025 rpcz_store.cc:276] 0426 13:20:02.788274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.813931 (+2025657us) service_pool.cc:225] Handling call
0426 13:20:04.814237 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.814865 (+ 628us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.815033 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.817190 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2072) took 2013 ms. Trace:
I20240426 13:20:04.817584 25025 rpcz_store.cc:276] 0426 13:20:02.804307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.816188 (+2011881us) service_pool.cc:225] Handling call
0426 13:20:04.816493 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.816845 (+ 352us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:04.816878 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:04.816941 (+ 63us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:04.816976 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.817035 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.817142 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.820348 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 898) took 9530 ms. Trace:
I20240426 13:20:04.820708 25025 rpcz_store.cc:276] 0426 13:19:55.289799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.818600 (+9528801us) service_pool.cc:225] Handling call
0426 13:20:04.820265 (+ 1665us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.828022 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 527) took 2223 ms. Trace:
I20240426 13:20:04.828578 25102 rpcz_store.cc:276] 0426 13:20:02.604396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.605324 (+ 928us) service_pool.cc:225] Handling call
0426 13:20:02.605629 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:02.606052 (+ 423us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:02.606457 (+ 405us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.616978 (+ 10521us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:02.646494 (+ 29516us) log.cc:842] Serialized 26 byte log entry
0426 13:20:02.646631 (+ 137us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:02.646672 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:02.646738 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.827750 (+2181012us) raft_consensus.cc:1671] finished
0426 13:20:04.827779 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.827957 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4646,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":6022,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":306,"thread_start_us":10010,"threads_started":1,"wal-append.queue_time_us":99405}
W20240426 13:20:04.833098 24996 leader_election.cc:334] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 1.984s (SENT)
I20240426 13:20:04.833647 24996 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:04.836297 27808 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240426 13:20:04.831671 27833 consensus_meta.cc:287] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.032s user 0.010s sys 0.010s
I20240426 13:20:04.838423 27833 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.839825 25030 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:04.842691 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2054) took 4968 ms. Trace:
I20240426 13:20:04.843055 25030 rpcz_store.cc:276] 0426 13:19:59.873952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.761711 (+4887759us) service_pool.cc:225] Handling call
0426 13:20:04.762436 (+ 725us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.840255 (+ 77819us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.842363 (+ 2108us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.842595 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177541888}
W20240426 13:20:04.846675 24996 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089): Timed out: RequestConsensusVote RPC to 127.23.61.3:38089 timed out after 2.135s (SENT)
I20240426 13:20:04.855610 27833 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:04.846915 25022 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.044s user 0.016s sys 0.015s
I20240426 13:20:04.856475 25022 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:04.856904 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:04.857069 25022 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2061) took 2324 ms (client timeout 1848 ms). Trace:
W20240426 13:20:04.857511 25022 rpcz_store.cc:269] 0426 13:20:02.532511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.691994 (+159483us) service_pool.cc:225] Handling call
0426 13:20:04.857000 (+2165006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.860881 25099 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:04.862046 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.862738 25099 raft_consensus.cc:2362] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4. Re-sending same reply.
I20240426 13:20:04.863669 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.877492 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.877771 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 382) took 6193 ms. Trace:
I20240426 13:20:04.878232 25031 rpcz_store.cc:276] 0426 13:19:58.684491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.572939 (+5888448us) service_pool.cc:225] Handling call
0426 13:20:04.666150 (+ 93211us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.667361 (+ 1211us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.877554 (+210193us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b54004169d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.877693 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213446912}
I20240426 13:20:04.880972 24996 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:04.881686 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0888649749998649s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20240426 13:20:04.881765 27725 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.881944 27808 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
W20240426 13:20:04.882257 25069 connection.cc:232] RPC call timeout handler was delayed by -1.0573649749999277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20240426 13:20:04.882726 27808 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:04.882498 27725 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.883872 27808 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.884738 27725 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.883678 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:04.887188 27725 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:04.887601 27725 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:04.887842 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.888053 27725 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:04.878810 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.912946 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2004) took 10134 ms. Trace:
I20240426 13:20:04.913379 25028 rpcz_store.cc:276] 0426 13:19:54.778183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:56.379732 (+1601549us) service_pool.cc:225] Handling call
0426 13:20:02.276248 (+5896516us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:02.380387 (+104139us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.912263 (+2531876us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400242554. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.912744 (+ 481us) spinlock_profiling.cc:243] Waited 5.87 s on lock 0x7b5400242550. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.912868 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13727666816}
I20240426 13:20:04.889070 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.915273 27630 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.915983 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1710) took 2088 ms. Trace:
I20240426 13:20:04.916435 25105 rpcz_store.cc:276] 0426 13:20:02.828014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.828156 (+ 142us) service_pool.cc:225] Handling call
0426 13:20:04.915887 (+2087731us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.917500 27707 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:04.917814 27911 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:04.917559 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.918908 27911 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.919343 27911 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.891914 27943 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:04.920609 25102 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:04.895315 27708 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.899121 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 510) took 5994 ms. Trace:
I20240426 13:20:04.871338 27833 consensus_queue.cc:1035] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:04.921528 27708 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:04.921473 25104 rpcz_store.cc:276] 0426 13:19:58.904992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.944441 (+ 39449us) service_pool.cc:225] Handling call
0426 13:19:58.945928 (+ 1487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.997074 (+ 51146us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:59.395389 (+398315us) log.cc:842] Serialized 14 byte log entry
0426 13:19:59.583331 (+187942us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:19:59.583379 (+ 48us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:19:59.583423 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.898639 (+5315216us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b540012d4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:04.898705 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.898921 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":223749,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":235,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":20037,"spinlock_wait_cycles":117074816,"thread_start_us":585608,"threads_started":2,"wal-append.queue_time_us":298248,"wal-append.run_cpu_time_us":11284,"wal-append.run_wall_time_us":1242894}
I20240426 13:20:04.923913 25024 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:04.924613 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1518) took 6317 ms. Trace:
I20240426 13:20:04.924937 25024 rpcz_store.cc:276] 0426 13:19:58.607628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.923332 (+6315704us) service_pool.cc:225] Handling call
0426 13:20:04.923721 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:04.924387 (+ 666us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.924564 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:04.923772 27708 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:04.925019 27911 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.929311 27911 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.918182 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.933243 27685 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.934247 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1942) took 5978 ms. Trace:
I20240426 13:20:04.933856 27685 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.934654 24951 rpcz_store.cc:276] 0426 13:19:58.955723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.955896 (+ 173us) service_pool.cc:225] Handling call
0426 13:19:59.263805 (+307909us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:59.291369 (+ 27564us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.933884 (+5642515us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54003c37d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:04.934070 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707353472}
I20240426 13:20:04.919631 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.936193 25616 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:04.936934 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.937142 27787 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:04.937462 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44512 (request call id 615) took 6571 ms. Trace:
I20240426 13:20:04.937628 25104 raft_consensus.cc:2362] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
I20240426 13:20:04.937791 27787 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.938623 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.938987 27725 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.937906 25620 rpcz_store.cc:276] 0426 13:19:58.366445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.594528 (+228083us) service_pool.cc:225] Handling call
0426 13:19:58.599727 (+ 5199us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:19:58.865940 (+266213us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:58.894253 (+ 28313us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.716675 (+3822422us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.828767 (+112092us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:04.313066 (+1484299us) log.cc:842] Serialized 26 byte log entry
0426 13:20:04.313276 (+ 210us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:04.313328 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:04.937076 (+623748us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:04.937131 (+ 55us) raft_consensus.cc:1671] finished
0426 13:20:04.937145 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:04.937300 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3408969,"prepare.queue_time_us":3351321,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":88,"raft.queue_time_us":234215,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":2672,"thread_start_us":713731,"threads_started":4,"wal-append.queue_time_us":997696,"wal-append.run_cpu_time_us":1441,"wal-append.run_wall_time_us":1003520}
W20240426 13:20:04.943086 25621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.162s user 0.017s sys 0.025s
W20240426 13:20:04.949126 25621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.168s user 0.018s sys 0.026s
I20240426 13:20:04.943722 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:04.919660 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:04.946139 24995 leader_election.cc:334] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.509s (SENT)
I20240426 13:20:04.949945 24995 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:04.950237 27725 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.950397 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.951146 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:04.951700 27855 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.952342 27911 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:20:04.952543 25104 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240426 13:20:04.952582 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.953166 25104 pending_rounds.cc:76] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 2
I20240426 13:20:04.953487 25104 pending_rounds.cc:97] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240426 13:20:04.953763 25104 raft_consensus.cc:2866] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.954308 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.955282 27451 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.955948 27451 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:04.957026 25104 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240426 13:20:04.957504 25104 pending_rounds.cc:76] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 4
I20240426 13:20:04.957808 25104 pending_rounds.cc:97] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240426 13:20:04.959317 25104 raft_consensus.cc:2866] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:04.965900 25104 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:04.966703 27451 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:04.965759 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.968475 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:04.969146 27451 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:04.972440 27725 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:04.973426 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:04.974117 25069 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:04.974725 25618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 584) took 2155 ms (client timeout 1716 ms). Trace:
I20240426 13:20:04.979532 27451 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240426 13:20:04.975306 25618 rpcz_store.cc:269] 0426 13:20:02.818848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.820929 (+ 2081us) service_pool.cc:225] Handling call
0426 13:20:04.973262 (+2152333us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:04.974629 (+ 1367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15843072}
I20240426 13:20:04.981367 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.980237 27725 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:04.982381 27911 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:04.983919 27911 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:04.985105 27868 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.984983 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.985548 27868 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.985677 27911 consensus_queue.cc:1046] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:04.986675 27868 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.986999 27376 raft_consensus.cc:3032] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:04.987350 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:04.987651 27376 raft_consensus.cc:729] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.987951 27868 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:04.988233 27911 raft_consensus.cc:2726] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:04.988634 27376 consensus_queue.cc:260] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:04.989761 27911 consensus_queue.cc:1046] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:04.990017 27376 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:04.990875 27911 raft_consensus.cc:3032] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:04.991274 27911 raft_consensus.cc:729] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:04.992376 27833 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.992949 27833 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:04.987093 27923 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:04.994728 25026 consensus_meta.cc:287] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 2.264s user 0.020s sys 0.002s
I20240426 13:20:04.995088 27833 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:04.995115 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:04.995734 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:04.996866 27633 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:04.997346 27633 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:04.998508 27986 log.cc:924] Time spent T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.414s user 0.006s sys 0.000s
I20240426 13:20:05.001477 27633 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:04.992373 27911 consensus_queue.cc:260] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.003137 27911 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.003753 27736 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.004901 27736 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:20:05.010109 24995 leader_election.cc:334] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.135s (SENT)
I20240426 13:20:05.010586 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.011432 27633 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240426 13:20:05.018030 24956 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1954) took 4737 ms (client timeout 1926 ms). Trace:
W20240426 13:20:05.018489 24956 rpcz_store.cc:269] 0426 13:20:00.280397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.280519 (+ 122us) service_pool.cc:225] Handling call
0426 13:20:05.017691 (+4737172us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b54000a7dd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.017831 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6806400}
I20240426 13:20:05.021842 27685 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:05.026795 24853 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60836 (request call id 174) took 23929 ms (client timeout 15000 ms). Trace:
W20240426 13:20:05.028889 24853 rpcz_store.cc:269] 0426 13:19:41.097084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:41.097474 (+ 390us) service_pool.cc:225] Handling call
0426 13:20:05.026581 (+23929107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:19:52.929618 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:19:53.108553 (+178935us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:19:53.108614 (+ 61us) write_op.cc:427] Acquired schema lock
0426 13:19:53.108634 (+ 20us) tablet.cc:660] Decoding operations
0426 13:19:53.109594 (+ 960us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:19:53.109691 (+ 97us) write_op.cc:633] Partition lock acquired for write op
0426 13:19:53.109721 (+ 30us) tablet.cc:683] Acquiring locks for 7 operations
0426 13:19:53.110452 (+ 731us) tablet.cc:699] Row locks acquired
0426 13:19:53.110491 (+ 39us) write_op.cc:260] PREPARE: finished
0426 13:19:53.110681 (+ 190us) write_op.cc:270] Start()
0426 13:19:53.110821 (+ 140us) write_op.cc:275] Timestamp: P: 1714137593110616 usec, L: 0
0426 13:19:53.110853 (+ 32us) op_driver.cc:331] REPLICATION: starting
0426 13:19:53.111549 (+ 696us) log.cc:842] Serialized 2793 byte log entry
0426 13:19:53.902042 (+790493us) op_driver.cc:444] REPLICATION: finished
0426 13:19:54.253470 (+351428us) write_op.cc:300] APPLY: starting
0426 13:19:54.303504 (+ 50034us) tablet.cc:1365] starting BulkCheckPresence
0426 13:19:54.341491 (+ 37987us) tablet.cc:1368] finished BulkCheckPresence
0426 13:19:54.399281 (+ 57790us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:19:54.868470 (+469189us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:19:54.871042 (+ 2572us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0426 13:19:54.871111 (+ 69us) write_op.cc:311] APPLY: finished
0426 13:19:54.871482 (+ 371us) log.cc:842] Serialized 59 byte log entry
0426 13:19:55.021765 (+150283us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:55.036844 (+ 15079us) write_op.cc:446] Released schema lock
0426 13:19:55.051202 (+ 14358us) write_op.cc:340] FINISH: Updating metrics
0426 13:19:55.079513 (+ 28311us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:19:55.137570 (+ 58057us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":283525,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":289210,"apply.run_cpu_time_us":27283,"apply.run_wall_time_us":939705,"num_ops":7,"prepare.queue_time_us":832991,"prepare.run_cpu_time_us":17746,"prepare.run_wall_time_us":288630,"raft.queue_time_us":307879,"raft.run_cpu_time_us":63993,"raft.run_wall_time_us":236014,"replication_time_us":791089,"thread_start_us":1302722,"threads_started":5,"wal-append.queue_time_us":539770,"wal-append.run_cpu_time_us":23292,"wal-append.run_wall_time_us":2532056}]]}
I20240426 13:20:05.037691 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.039086 27755 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:05.039484 27755 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.039820 27755 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:05.045380 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 55921.6us
I20240426 13:20:05.044438 27932 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:05.048107 27633 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:05.052857 25096 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:05.061383 27955 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:05.071065 25622 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:05.071652 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.072335 25104 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:05.073088 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:05.073673 25622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 585) took 2254 ms (client timeout 1824 ms). Trace:
I20240426 13:20:05.074052 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.074853 25104 raft_consensus.cc:2379] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:05.077968 27974 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.078395 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.078578 27974 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.080214 27633 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:05.080678 27633 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.081028 27633 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.081406 25027 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:05.082043 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1515) took 6635 ms. Trace:
I20240426 13:20:05.082355 25027 rpcz_store.cc:276] 0426 13:19:58.446335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.080820 (+6634485us) service_pool.cc:225] Handling call
0426 13:20:05.081204 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.081804 (+ 600us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.081980 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:05.074085 25622 rpcz_store.cc:269] 0426 13:20:02.819637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.819776 (+ 139us) service_pool.cc:225] Handling call
0426 13:20:05.073513 (+2253737us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54004a24d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.073609 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5167418752}
W20240426 13:20:05.096761 25616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 619) took 5563 ms (client timeout 2123 ms). Trace:
W20240426 13:20:05.097301 25616 rpcz_store.cc:269] 0426 13:19:59.532913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:59.620244 (+ 87331us) service_pool.cc:225] Handling call
0426 13:20:05.096490 (+5476246us) spinlock_profiling.cc:243] Waited 5.29 s on lock 0x7b540065f754. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.096678 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12177081088}
I20240426 13:20:05.101161 27787 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:05.099421 27633 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.102736 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.126399 27869 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:05.127151 27974 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.127782 27869 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.128206 27869 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.129089 27869 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.138463 25029 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20240426 13:20:05.144704 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.145547 25096 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.146926 25103 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:05.147310 25103 pending_rounds.cc:76] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:20:05.147568 25103 pending_rounds.cc:97] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:05.147835 25103 raft_consensus.cc:2866] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:05.148147 26679 consensus_queue.cc:1035] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240426 13:20:05.151104 25069 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.926s (SENT)
I20240426 13:20:05.150902 26679 consensus_queue.cc:1046] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240426 13:20:05.152889 25070 connection.cc:232] RPC call timeout handler was delayed by -1.6116430530000798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.209s for a response
W20240426 13:20:05.153472 25070 connection.cc:232] RPC call timeout handler was delayed by -1.4123543020000398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18480000000000002s for a response
W20240426 13:20:05.153867 25070 connection.cc:232] RPC call timeout handler was delayed by -1.3377743649999729s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20240426 13:20:05.154258 25070 connection.cc:232] RPC call timeout handler was delayed by -1.1357782520000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
I20240426 13:20:05.125743 27755 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.161828 27808 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:05.162470 27808 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.164171 27943 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.164130 27932 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:05.137924 27633 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.168792 25103 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:20:05.170583 27755 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.163000 27855 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.164649 27808 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.181540 27855 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.183615 27855 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.164827 27943 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.185623 27943 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.186586 27911 raft_consensus.cc:3032] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:05.186965 27911 raft_consensus.cc:729] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.187867 27911 consensus_queue.cc:260] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.189056 27911 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:05.190026 25026 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1513) took 6964 ms (client timeout 2473 ms). Trace:
W20240426 13:20:05.190554 25026 rpcz_store.cc:269] 0426 13:19:58.225260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.346301 (+121041us) service_pool.cc:225] Handling call
0426 13:20:05.189732 (+6843431us) spinlock_profiling.cc:243] Waited 4.02 s on lock 0x7b5400242a54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.189932 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9252995328}
I20240426 13:20:05.193713 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.196517 25104 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.180074 27869 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.203752 27755 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
W20240426 13:20:05.227371 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 60767.3us
I20240426 13:20:05.228612 25070 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.233186 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.239099 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.242851 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.243352 25026 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.246198 27946 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:05.246639 25024 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.247502 25070 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.248605 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.249548 26679 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:05.250053 26679 raft_consensus.cc:686] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:05.250113 27978 log.cc:924] Time spent T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 2.549s user 0.001s sys 0.002s
I20240426 13:20:05.250960 26679 consensus_queue.cc:237] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.252249 27946 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:05.256068 27868 consensus_queue.cc:1046] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:05.247799 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.255885 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.260648 25070 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:05.261134 25617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 583) took 2443 ms (client timeout 2047 ms). Trace:
W20240426 13:20:05.261569 25617 rpcz_store.cc:269] 0426 13:20:02.818203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.820748 (+ 2545us) service_pool.cc:225] Handling call
0426 13:20:05.255583 (+2434835us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:05.261041 (+ 5458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3658022784}
I20240426 13:20:05.262054 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2043) took 5866 ms. Trace:
I20240426 13:20:05.262516 25029 rpcz_store.cc:276] 0426 13:19:59.395940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:04.719847 (+5323907us) service_pool.cc:225] Handling call
0426 13:20:04.728916 (+ 9069us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.139931 (+411015us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.261792 (+121861us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54003f6e54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:05.261978 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":941240704}
I20240426 13:20:05.263521 27653 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:05.264079 27653 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.264961 27653 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.266865 27376 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.268105 27376 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:05.268487 27376 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.269366 27376 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.270499 27376 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.275759 25100 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:05.258952 27911 raft_consensus.cc:3032] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.284905 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.286453 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.296551 25026 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:05.297653 25029 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.302390 24852 catalog_manager.cc:5484] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.303488 27725 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:20:05.305085 27725 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.305262 24852 catalog_manager.cc:5484] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:05.309536 25070 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.310321 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.310428 27725 consensus_queue.cc:1046] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:05.310938 25098 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:05.311497 27498 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:05.312016 27498 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.313030 27498 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.316592 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.317559 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.318781 27755 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:05.319177 27755 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.319506 27755 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:05.311827 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.322110 25101 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:05.324354 27633 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:05.325430 27633 raft_consensus.cc:3032] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:05.287842 27911 raft_consensus.cc:729] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.325830 27633 raft_consensus.cc:729] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.330423 27653 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:05.332816 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 520) took 2946 ms. Trace:
I20240426 13:20:05.333240 25100 rpcz_store.cc:276] 0426 13:20:02.387429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.409006 (+ 21577us) service_pool.cc:225] Handling call
0426 13:20:02.412809 (+ 3803us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.276203 (+2863394us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.332600 (+ 56397us) spinlock_profiling.cc:243] Waited 2.86 s on lock 0x7b5400384ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:05.332761 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6583452544}
I20240426 13:20:05.330220 27633 consensus_queue.cc:260] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.337649 25104 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.332687 25070 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.340590 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.338274 25096 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:05.307894 24852 catalog_manager.cc:5484] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.341905 27633 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.341753 27725 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:05.346457 27376 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:05.346628 27725 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.347061 27376 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.347528 27376 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.347775 27725 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.327239 27911 consensus_queue.cc:260] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.352792 27911 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:05.352859 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.353574 25031 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:05.356918 27930 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:05.357472 27930 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.357487 24995 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.294128 27916 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.363126 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.363895 25616 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:05.363536 24852 catalog_manager.cc:5484] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:05.365207 24852 catalog_manager.cc:5484] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.367417 27808 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:05.367985 27808 raft_consensus.cc:686] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.370411 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.369985 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.371362 25099 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.369038 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.369551 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.377697 25622 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.367188 24852 catalog_manager.cc:5484] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:05.387913 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 91876.4us
I20240426 13:20:05.371060 27808 consensus_queue.cc:237] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.390460 27376 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.394599 24996 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.395898 25620 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.369552 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.397797 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.423811 25623 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:05.400329 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.400439 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.425230 25024 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.425733 25023 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:05.384675 24852 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 5 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.422791 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.427739 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.427073 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 415) took 4829 ms. Trace:
I20240426 13:20:05.380806 27916 raft_consensus.cc:686] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.429246 24996 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.430850 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2023) took 6822 ms. Trace:
I20240426 13:20:05.432358 27438 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:05.432730 27955 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.432686 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49046 (request call id 27) took 2922 ms. Trace:
I20240426 13:20:05.381940 27930 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:05.430423 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.435562 25023 rpcz_store.cc:276] 0426 13:19:58.608194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.392200 (+6784006us) service_pool.cc:225] Handling call
0426 13:20:05.418700 (+ 26500us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:05.427684 (+ 8984us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.430780 (+ 3096us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:05.437548 25097 rpcz_store.cc:276] 0426 13:20:02.504248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.504744 (+ 496us) service_pool.cc:225] Handling call
0426 13:20:05.423918 (+2919174us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90311d53 00007fda9525c48a 00007fda8e94bcd4 00007fda8e94bc62 00007fda8e94bb91 00007fda8e94bb0c 00007fda8e94adc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:05.424026 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":577234176}
I20240426 13:20:05.441754 25104 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:05.443009 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.444612 27932 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:05.443586 25622 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:05.452867 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.433465 27946 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:05.453737 25623 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:05.476739 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.445088 27932 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.486217 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.449548 27438 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.452457 27498 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:05.451961 27955 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:05.429831 25615 rpcz_store.cc:276] 0426 13:19:58.999038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:58.999183 (+ 145us) service_pool.cc:225] Handling call
0426 13:19:58.999556 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:19:58.999770 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:19:59.000132 (+ 362us) log.cc:842] Serialized 14 byte log entry
0426 13:19:59.211809 (+211677us) log.cc:842] Serialized 14 byte log entry
0426 13:20:02.716114 (+3504305us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda941c17a8 00007fda8d3ecff6 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34
0426 13:20:02.769975 (+ 53861us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:03.158454 (+388479us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:03.176169 (+ 17715us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:03.310599 (+134430us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:03.348324 (+ 37725us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:03.828055 (+479731us) inbound_call.cc:173] Queueing success response
0426 13:20:04.704986 (+876931us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b18001f1800. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
Metrics: {"mutex_wait_us":52914,"prepare.queue_time_us":27128,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":191214,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":13045,"spinlock_wait_cycles":61451776,"thread_start_us":211881,"threads_started":2,"wal-append.queue_time_us":391092,"wal-append.run_cpu_time_us":5950,"wal-append.run_wall_time_us":1230833}
I20240426 13:20:05.495637 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:05.461103 27946 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.501936 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.502218 27438 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.503094 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.503166 24852 catalog_manager.cc:5484] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 2 to 4, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:05.504405 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 5.22676e+06us
I20240426 13:20:05.505574 27376 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.489594 27932 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.496659 27498 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:05.507689 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080003bae0 after lost signal to thread 23799
W20240426 13:20:05.522996 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003984e0 after lost signal to thread 24888
W20240426 13:20:05.523711 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080034e5e0 after lost signal to thread 24920
I20240426 13:20:05.539324 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:05.538980 24852 catalog_manager.cc:5484] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:05.540700 24852 catalog_manager.cc:5484] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 reported cstate change: term changed from 1 to 3, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:05.543195 25024 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:05.536971 27498 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:05.548542 27955 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.550396 25030 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:05.550688 25070 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.555608 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:05.557159 25023 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.497879 27810 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20240426 13:20:05.561286 27794 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240426 13:20:05.570310 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 207616us
W20240426 13:20:05.577256 28007 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240426 13:20:05.586108 27653 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:05.586558 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:05.586778 27653 raft_consensus.cc:686] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
W20240426 13:20:05.588348 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800333600 after lost signal to thread 24980
I20240426 13:20:05.587796 27653 consensus_queue.cc:237] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.588671 27310 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:05.589591 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800407220 after lost signal to thread 25055
I20240426 13:20:05.536711 27916 consensus_queue.cc:237] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.590894 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003a8a80 after lost signal to thread 25129
I20240426 13:20:05.591321 27696 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:05.591786 27696 raft_consensus.cc:686] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:05.569316 27946 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.593580 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003ef3a0 after lost signal to thread 25650
W20240426 13:20:05.594512 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080033f000 after lost signal to thread 27631
I20240426 13:20:05.589128 27310 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:05.597186 27946 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:05.597666 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08000395c0 after lost signal to thread 27640
W20240426 13:20:05.602329 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003eeae0 after lost signal to thread 27684
W20240426 13:20:05.602563 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080029f020 after lost signal to thread 27686
W20240426 13:20:05.597507 27376 consensus_queue.cc:1159] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 7 status { last_received { term: 6 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20240426 13:20:05.602726 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080030f180 after lost signal to thread 27687
W20240426 13:20:05.604730 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7180 after lost signal to thread 27710
I20240426 13:20:05.606659 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.609462 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.607404 25029 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
W20240426 13:20:05.622882 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800333ae0 after lost signal to thread 27717
I20240426 13:20:05.624084 25099 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
W20240426 13:20:05.624541 25589 connection.cc:232] RPC call timeout handler was delayed by -1.2860821720000786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20240426 13:20:05.626067 25589 connection.cc:232] RPC call timeout handler was delayed by -1.1427117030000318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20240426 13:20:05.629307 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:05.630040 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.632172 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:05.639016 26679 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:05.639384 26679 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:05.639591 25069 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:05.639647 26679 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:05.644680 25104 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:05.649919 25031 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:05.604562 27755 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.642468 27833 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:05.592666 27696 consensus_queue.cc:237] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:05.642606 27725 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240426 13:20:05.650455 27868 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:05.702587 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.703637 25096 raft_consensus.cc:2379] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 5: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:05.703761 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.704463 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:05.706326 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.664060 25616 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20240426 13:20:05.664363 27310 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:05.664762 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.708513 25617 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:05.667510 27410 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:05.668320 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:05.716696 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:05.699708 27868 raft_consensus.cc:686] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.718140 27868 consensus_queue.cc:237] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.670718 27755 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.674782 25105 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:05.667495 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 416) took 5385 ms. Trace:
I20240426 13:20:05.720525 25099 raft_consensus.cc:595] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:05.720990 25099 raft_consensus.cc:603] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:05.720475 25621 rpcz_store.cc:276] 0426 13:20:00.282461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:00.283457 (+ 996us) service_pool.cc:225] Handling call
0426 13:20:00.283802 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:00.284036 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:02.780848 (+2496812us) log.cc:842] Serialized 14 byte log entry
0426 13:20:05.652161 (+2871313us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:05.659751 (+ 7590us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:05.667183 (+ 7432us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.667283 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:05.667436 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3126256,"prepare.queue_time_us":2433729,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":694615,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":6070,"thread_start_us":2227457,"threads_started":2,"wal-append.queue_time_us":1711245,"wal-append.run_cpu_time_us":1743,"wal-append.run_wall_time_us":1016151}
I20240426 13:20:05.687990 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.694236 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:05.727073 25098 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:20:05.727108 25100 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20240426 13:20:05.698642 27932 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.732781 25100 pending_rounds.cc:76] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 3
I20240426 13:20:05.741294 25100 pending_rounds.cc:97] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:05.741643 25100 raft_consensus.cc:2866] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:05.743767 25098 raft_consensus.cc:2454] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:05.747489 25023 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:05.749511 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:05.760416 24996 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:05.773028 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 146093us
I20240426 13:20:05.772569 27869 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:05.801821 27869 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:05.806922 25023 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:05.802788 26679 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.794723 27794 raft_consensus.cc:2781] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:05.838176 27498 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:05.869966 28019 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
W20240426 13:20:05.927487 28014 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.369s user 0.002s sys 0.002s
I20240426 13:20:05.975881 25104 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 4. Ops: []
I20240426 13:20:05.976931 25624 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 587) took 3150 ms. Trace:
I20240426 13:20:05.988502 25624 rpcz_store.cc:276] 0426 13:20:02.826410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:02.826658 (+ 248us) service_pool.cc:225] Handling call
0426 13:20:04.853461 (+2026803us) spinlock_profiling.cc:243] Waited 871 us on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b53ee 00000000003399f9 00007fda951e4570 00007fda952aa294 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:04.853657 (+ 196us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.603300 (+749643us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:05.603340 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:05.603832 (+ 492us) log.cc:842] Serialized 26 byte log entry
0426 13:20:05.784734 (+180902us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:05.784774 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:05.784851 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:05.976599 (+191748us) raft_consensus.cc:1671] finished
0426 13:20:05.976619 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:05.976873 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":749199,"raft.queue_time_us":749835,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":2004352,"thread_start_us":180469,"threads_started":1,"wal-append.queue_time_us":206095}
I20240426 13:20:05.743674 27725 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
I20240426 13:20:05.991575 28035 raft_consensus.cc:1050] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8: attempting to promote NON_VOTER 1590f80bdade41c6a755b8501c55fe16 to VOTER
I20240426 13:20:05.991880 25025 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:05.991465 27708 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:05.993003 27708 consensus_queue.cc:1225] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:20:05.802781 27869 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:05.979784 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:05.995262 25622 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:05.996326 27932 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.003371 25622 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:05.819640 27916 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20240426 13:20:06.004671 25069 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:05.983271 26679 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:05.808017 27955 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:20:06.005800 26679 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader election won for term 4
I20240426 13:20:05.994001 28035 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.009017 27794 raft_consensus.cc:686] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:06.010110 27794 consensus_queue.cc:237] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.006297 26679 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:06.014417 26679 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.020117 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:06.020812 25621 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.023114 25098 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:06.025172 27932 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:06.031261 25621 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:06.032483 24995 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:06.033182 27498 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.033525 27932 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:06.034005 25098 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240426 13:20:06.034735 28033 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 13:20:06.040607 27749 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.041373 27766 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.041241 27749 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.041968 27766 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.043334 27749 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.043746 27766 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.043934 27749 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.057123 27775 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:06.044153 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800113620 after lost signal to thread 27753
W20240426 13:20:06.068588 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800344bc0 after lost signal to thread 27756
W20240426 13:20:06.068979 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080035caa0 after lost signal to thread 27768
W20240426 13:20:06.069301 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800339a80 after lost signal to thread 27771
I20240426 13:20:06.068055 27775 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.052662 27881 raft_consensus.cc:659] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } } [suppressed 5 similar messages]
I20240426 13:20:06.069562 27410 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:06.070355 27410 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.067677 27749 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.072126 27928 raft_consensus.cc:686] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.072582 27749 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:06.073475 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080040a280 after lost signal to thread 27811
W20240426 13:20:06.073836 24885 debug-util.cc:398] Leaking SignalData structure 0x7b0800249900 after lost signal to thread 27813
W20240426 13:20:06.074465 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003e2dc0 after lost signal to thread 27838
I20240426 13:20:06.073513 27928 consensus_queue.cc:237] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.074726 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08003341c0 after lost signal to thread 27841
W20240426 13:20:06.075383 24885 debug-util.cc:398] Leaking SignalData structure 0x7b080039bea0 after lost signal to thread 27844
W20240426 13:20:06.075719 24885 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4aa0 after lost signal to thread 27849
I20240426 13:20:06.076198 27900 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.076541 28036 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:06.076961 28036 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:06.076678 27900 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.051348 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1728) took 1189 ms. Trace:
I20240426 13:20:06.081954 25105 rpcz_store.cc:276] 0426 13:20:04.862125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.369293 (+507168us) service_pool.cc:225] Handling call
0426 13:20:06.038290 (+668997us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54003ee4d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:06.051254 (+ 12964us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678655744}
W20240426 13:20:06.141609 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 331370us
I20240426 13:20:06.063899 25621 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.146435 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:06.115307 28026 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20240426 13:20:06.055716 27833 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:06.147852 25100 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.153602 27770 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.154163 27770 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.156296 28036 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.157714 28036 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:05.991484 25617 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.147361 25621 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:06.161648 27833 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:06.145128 27932 raft_consensus.cc:686] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:06.139770 27755 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20240426 13:20:06.181589 25022 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:06.175024 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:06.191511 24956 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20240426 13:20:06.191586 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.192168 27730 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:06.192641 27730 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.193002 27730 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:06.193279 27775 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.166242 27410 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.194487 25621 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.195364 27410 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.183265 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.196154 27637 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.196280 24947 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.196736 27637 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:06.198310 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49040.4us
I20240426 13:20:06.198805 27637 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.186604 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.187315 27900 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.200860 23796 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240426 13:20:06.202986 27736 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:06.203428 23796 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240426 13:20:06.204151 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:06.204319 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 665065us
I20240426 13:20:06.204831 24952 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.186515 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.193104 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.188143 27310 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:06.188273 24920 connection.cc:232] RPC call timeout handler was delayed by -4.73400823999998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
I20240426 13:20:06.205812 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:06.171640 27770 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.209254 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:06.219538 25100 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:06.223131 24947 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:06.206110 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.195835 25621 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.221372 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.205770 27736 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.241163 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.186688 28025 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.240465 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.245276 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.252454 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:06.250501 24952 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:06.272585 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 66851.1us
I20240426 13:20:06.207983 27647 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.279875 25028 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:06.282709 27759 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:06.282812 27932 consensus_queue.cc:237] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.283799 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:06.284557 25023 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.289134 25031 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:20:06.285771 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:06.284765 27810 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:06.292531 25026 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:06.292461 27810 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.293866 25025 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:06.294505 27810 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.277638 27725 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20240426 13:20:06.296344 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.298688 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:06.245793 27770 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.299849 27914 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:06.220561 27964 consensus_queue.cc:1046] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20240426 13:20:06.240664 27736 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:06.301548 27410 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:06.302557 27736 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.303745 27736 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.285768 27310 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.273473 27853 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:06.300225 27881 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:06.284876 25622 raft_consensus.cc:1261] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240426 13:20:06.304126 27647 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.304600 27914 consensus_queue.cc:1035] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:06.305445 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.559173 24949 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
W20240426 13:20:06.304284 27410 consensus_queue.cc:1159] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 6 status { last_received { term: 6 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 6 index: 8 } }
I20240426 13:20:06.306015 27271 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:06.307265 27880 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:06.308751 24947 raft_consensus.cc:1261] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:06.443259 27382 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.093s
I20240426 13:20:06.309973 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.310645 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.311693 24956 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240426 13:20:06.300262 27770 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.314159 28025 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.415686 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:06.575459 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 159856us
W20240426 13:20:06.403496 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 91085us
I20240426 13:20:06.413038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.480723 25623 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:06.576352 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:06.326064 27881 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:06.421896 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 105090us
I20240426 13:20:06.407497 25031 raft_consensus.cc:1261] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240426 13:20:06.396458 25023 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:06.412212 27725 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.232981 27451 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.057s
I20240426 13:20:06.301587 27964 raft_consensus.cc:3032] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:06.325775 27853 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:06.443246 27653 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:06.443691 26679 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.441848 27350 consensus_queue.cc:1046] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
W20240426 13:20:06.412366 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 121347us
I20240426 13:20:06.597591 27881 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.309334 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.660992 24948 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:06.580642 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.568142 24950 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:06.583163 27946 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.576867 24955 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.495672 27730 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.627753 24953 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:06.627264 24954 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:06.628185 25618 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:20:06.628832 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.632158 27757 raft_consensus.cc:2741] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:20:06.583736 27923 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:06.608628 27964 raft_consensus.cc:729] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:06.597404 28027 log.cc:924] Time spent T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20240426 13:20:06.567343 27916 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.618085 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.651072 26679 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.403272 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.656976 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.638303 27853 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:06.664371 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 56021us
I20240426 13:20:06.664681 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.662257 24947 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.668278 24920 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:20:06.675928 25623 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240426 13:20:06.637780 28025 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.677511 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:06.677778 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 456434us
I20240426 13:20:06.678263 24949 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
W20240426 13:20:06.686618 24995 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.035s sys 0.047s
I20240426 13:20:06.609207 27725 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.586398 27271 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.672318 27946 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.692363 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:06.695809 24995 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s user 0.035s sys 0.047s
I20240426 13:20:06.701265 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.603446 27833 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.705233 24956 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:06.709131 26679 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.645576 27707 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
W20240426 13:20:06.704747 28044 log.cc:924] Time spent T 00000000000000000000000000000000 P e2ab89ff7e18462eb10e19320ff8beaa: Append to log took a long time: real 0.142s user 0.000s sys 0.002s
I20240426 13:20:06.707888 24955 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:06.709772 26679 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240426 13:20:06.713550 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 62891.7us
I20240426 13:20:06.719221 27350 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:06.719610 27350 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:06.670032 27911 consensus_queue.cc:1035] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20240426 13:20:06.752347 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.756238 25623 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:06.711803 27630 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.027s sys 0.057s
I20240426 13:20:06.719749 27647 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.766623 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.760701 27637 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:06.768923 27633 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.770488 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.772181 27766 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:06.773157 27964 consensus_queue.cc:260] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:06.773941 28037 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.585s user 0.001s sys 0.003s
I20240426 13:20:06.720211 27852 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.100s
I20240426 13:20:06.781795 27946 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.708495 27916 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.786082 28041 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20240426 13:20:06.719961 27350 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:06.788897 25616 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:06.897166 27730 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:06.791290 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.792562 24950 raft_consensus.cc:3032] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:06.936188 24950 raft_consensus.cc:729] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.793345 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:06.802044 27725 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:06.802801 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.804699 27271 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:06.807652 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 49452.3us
I20240426 13:20:06.829556 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:06.832739 25097 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.021s sys 0.010s
I20240426 13:20:06.713129 27869 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20240426 13:20:06.809988 27810 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20240426 13:20:06.860411 24995 leader_election.cc:334] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.514s (SENT)
W20240426 13:20:06.846199 25621 consensus_meta.cc:287] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.605s user 0.011s sys 0.000s
I20240426 13:20:06.893216 24955 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:06.850281 27630 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.644s user 0.029s sys 0.057s
I20240426 13:20:06.847055 27770 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:06.892172 27911 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:06.894851 27932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.023s sys 0.019s
I20240426 13:20:06.790535 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.935678 24949 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:06.720345 28034 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.259s
W20240426 13:20:06.932606 27808 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.506s user 0.006s sys 0.032s
I20240426 13:20:06.944152 25618 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:06.939819 24950 consensus_queue.cc:260] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.977262 24950 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:06.946230 25097 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.021s sys 0.010s
I20240426 13:20:06.941371 27946 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.951249 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:06.979686 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 119335us
I20240426 13:20:06.980621 24951 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Stepping down as leader of term 4
W20240426 13:20:06.951803 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 145083us
I20240426 13:20:06.981073 24951 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.982084 24951 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:06.983476 24951 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:06.983812 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:06.984385 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:06.985152 24949 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:06.985374 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:06.986678 24950 raft_consensus.cc:1261] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240426 13:20:06.987035 24950 pending_rounds.cc:76] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:06.987238 24950 pending_rounds.cc:97] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240426 13:20:06.899173 27633 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:06.989351 27633 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:06.954993 28045 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240426 13:20:06.991760 25103 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:06.899081 27637 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:06.961638 27932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.023s sys 0.019s
I20240426 13:20:06.937878 27833 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.054036 27916 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.054941 27310 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.059954 27677 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:07.060524 27677 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:07.061486 27677 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.062647 25624 raft_consensus.cc:3032] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:07.063112 25624 raft_consensus.cc:729] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:07.063298 27677 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.064047 25624 consensus_queue.cc:260] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.064714 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.065333 25624 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:07.065726 25103 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:07.066987 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.972846 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.068346 24947 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:06.927948 27766 consensus_queue.cc:1225] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:06.987183 27900 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:06.991581 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.072746 27900 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:06.992370 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.073174 27900 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.074259 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.075390 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.993661 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.075953 25030 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.076797 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.076936 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.993903 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.994230 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.077795 25099 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.078348 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.078611 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.078956 25031 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.079253 25099 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:06.994793 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.080802 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.081588 25098 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:07.082293 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.083264 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:06.995018 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.995234 27759 raft_consensus.cc:3032] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:06.995720 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.996596 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:06.996577 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.994259 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.048928 25621 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:06.995632 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.997931 24950 raft_consensus.cc:2866] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.096495 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.001003 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.097441 25022 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.004560 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.013250 28025 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:07.029888 24954 raft_consensus.cc:3032] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:06.992370 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:06.946614 25617 raft_consensus.cc:2396] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:06.952877 25615 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 598) took 1258 ms. Trace:
I20240426 13:20:07.039281 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.959764 27730 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:07.112758 25024 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:07.113114 27730 raft_consensus.cc:686] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.113680 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.114318 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:07.114061 27730 consensus_queue.cc:237] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.115171 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.115852 25024 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.116137 27916 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.121105 27916 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.122179 27637 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.123030 27916 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.073096 25023 raft_consensus.cc:2379] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.123445 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.124181 24955 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.124291 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.124887 25023 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.124948 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.125763 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.126067 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.126842 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.127105 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.127686 24955 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:07.128232 25023 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 4. Current term is 5. Ops: []
W20240426 13:20:07.131027 28036 consensus_meta.cc:287] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.973s user 0.011s sys 0.000s
W20240426 13:20:07.087091 25029 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.000s sys 0.067s
I20240426 13:20:07.089098 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.131922 25104 raft_consensus.cc:1261] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20240426 13:20:07.132165 25099 raft_consensus.cc:2396] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:20:07.132283 25104 pending_rounds.cc:76] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 8
I20240426 13:20:07.132620 25104 pending_rounds.cc:97] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20240426 13:20:07.132848 25104 raft_consensus.cc:2866] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.133136 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.133723 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:07.093124 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.093953 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.135075 25101 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.135110 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.136008 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.136834 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.095063 25026 raft_consensus.cc:2379] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.137239 25101 raft_consensus.cc:1203] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[7.4-7.4] Dedup: 7.4->[]
I20240426 13:20:07.051376 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:07.139889 25029 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.823s user 0.000s sys 0.068s
I20240426 13:20:07.141145 25105 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:07.134572 25025 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:07.116801 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.117616 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:06.979641 27943 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:07.118611 28035 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.120162 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.077473 25030 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.040572 27946 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.098745 24954 raft_consensus.cc:729] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.137696 25101 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20240426 13:20:07.111588 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.089949 27759 raft_consensus.cc:729] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:06.975147 27911 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.158757 25028 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.136022 24953 raft_consensus.cc:2396] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240426 13:20:07.138960 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.120177 25624 raft_consensus.cc:1261] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:07.099558 25615 rpcz_store.cc:276] 0426 13:20:05.687392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:05.687788 (+ 396us) service_pool.cc:225] Handling call
0426 13:20:05.688133 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:05.809598 (+121465us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:05.809629 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:05.817033 (+ 7404us) log.cc:842] Serialized 26 byte log entry
0426 13:20:06.117256 (+300223us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:06.117314 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:06.117405 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:06.773313 (+655908us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:06.945005 (+171692us) raft_consensus.cc:1671] finished
0426 13:20:06.945032 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:06.945295 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":138159,"raft.queue_time_us":121926,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4460,"spinlock_wait_cycles":14870912,"thread_start_us":135554,"threads_started":1,"wal-append.queue_time_us":214630}
I20240426 13:20:07.144770 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.148775 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 492363us
I20240426 13:20:07.155815 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.164494 25070 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.098575 28025 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.240782 28025 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.240689 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.242094 28025 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.242321 25027 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:07.243216 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.243783 25027 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:07.091172 27881 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.244519 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.245079 25027 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:07.245779 27881 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.246300 27725 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:07.246942 27725 raft_consensus.cc:686] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:07.247809 27725 consensus_queue.cc:237] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.248521 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.249395 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.250159 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.196292 27943 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.156899 28035 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.208472 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:07.209385 27653 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.215471 25105 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:07.216430 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.218343 27451 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.154685 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 602) took 1091 ms. Trace:
I20240426 13:20:07.274113 24953 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.236660 27946 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.237329 28036 consensus_queue.cc:1035] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20240426 13:20:07.164193 27916 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.252077 25027 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:07.165900 24954 consensus_queue.cc:260] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.262164 25096 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.273872 27869 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.215654 24949 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:07.302824 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.303550 24949 raft_consensus.cc:2362] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
W20240426 13:20:07.292069 24956 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.537s user 0.011s sys 0.000s
I20240426 13:20:07.252755 27647 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:07.255112 25100 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:07.305648 25031 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:07.297930 24954 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.299388 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.300475 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.263525 27943 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.305485 24956 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:07.309280 24955 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.310585 25588 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.036s sys 0.049s
I20240426 13:20:07.312119 24953 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:07.312105 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 1994) took 1004 ms. Trace:
I20240426 13:20:07.315244 24956 rpcz_store.cc:276] 0426 13:20:06.307242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.589369 (+282127us) service_pool.cc:225] Handling call
0426 13:20:07.312024 (+722655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:07.316526 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.316731 24947 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:07.317772 25588 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s user 0.037s sys 0.049s
I20240426 13:20:07.304477 27271 raft_consensus.cc:2741] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20240426 13:20:07.300637 27946 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.203920 27637 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.098s
I20240426 13:20:07.318915 24956 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.317656 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.302182 27869 raft_consensus.cc:482] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:07.324894 25588 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.752s (SENT) [suppressed 42 similar messages]
I20240426 13:20:07.324412 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.328553 24949 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:07.319226 27943 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.306349 27647 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:07.351254 24953 raft_consensus.cc:1226] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:07.293874 27653 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.294056 27451 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.353181 25069 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.027s sys 0.017s
I20240426 13:20:07.289695 25621 rpcz_store.cc:276] 0426 13:20:06.063115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.063286 (+ 171us) service_pool.cc:225] Handling call
0426 13:20:06.063732 (+ 446us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.095356 (+1031624us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.150110 (+ 54754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:07.325914 27946 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.363692 24956 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:07.361788 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.308715 27755 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20240426 13:20:07.361851 25069 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.027s sys 0.018s
I20240426 13:20:07.330984 27869 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.307807 27916 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.391448 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 696476us
I20240426 13:20:07.405082 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.363015 27382 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.033s sys 0.018s
W20240426 13:20:07.360311 27928 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.017s sys 0.027s
W20240426 13:20:07.363440 27708 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.014s sys 0.037s
W20240426 13:20:07.413455 27868 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.046s sys 0.046s
I20240426 13:20:07.420269 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.389602 27946 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.363631 27633 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.302385 27759 consensus_queue.cc:260] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.425172 25589 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.014s sys 0.079s
I20240426 13:20:07.412431 25098 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:07.425776 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.440822 25589 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.015s sys 0.080s
W20240426 13:20:07.441175 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 843699us
I20240426 13:20:07.428130 25099 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
W20240426 13:20:07.413918 27382 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s user 0.034s sys 0.018s
I20240426 13:20:07.441815 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.454239 26679 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:07.422899 27928 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.735s user 0.017s sys 0.027s
W20240426 13:20:07.424082 27708 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.786s user 0.015s sys 0.037s
I20240426 13:20:07.459537 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:07.424695 27868 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.843s user 0.047s sys 0.046s
I20240426 13:20:07.451602 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.461503 26679 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.391541 28036 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20240426 13:20:07.497994 26679 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.502125 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.460465 27914 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:07.509696 25100 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:07.460122 28034 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:07.481550 27911 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:07.513084 24948 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.029s sys 0.025s
W20240426 13:20:07.514889 24948 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.795s user 0.030s sys 0.025s
I20240426 13:20:07.501626 27759 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:07.521831 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.524888 24953 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:07.526717 25100 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
W20240426 13:20:07.528357 28050 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20240426 13:20:07.481245 27647 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.516865 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:07.537942 27869 raft_consensus.cc:504] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.513059 27853 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.805s user 0.008s sys 0.004s
I20240426 13:20:07.544430 27853 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.546838 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.820s user 0.014s sys 0.080s
W20240426 13:20:07.547253 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.938s user 0.014s sys 0.080s
I20240426 13:20:07.548167 27853 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.449855 27633 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:07.594923 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.595840 24953 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:07.552201 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44737.3us
I20240426 13:20:07.597632 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.598954 27271 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.599463 27271 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:07.599874 27271 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.601785 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 84838.3us
I20240426 13:20:07.549886 28035 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.619189 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.620085 25100 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:07.630492 24951 consensus_meta.cc:287] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.646s user 0.014s sys 0.000s
I20240426 13:20:07.636766 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.523905 27382 consensus_queue.cc:1046] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20240426 13:20:07.650687 25028 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:07.600268 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:07.664662 25105 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
W20240426 13:20:07.670344 24997 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.000s sys 0.016s
I20240426 13:20:07.670424 27869 leader_election.cc:288] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.671192 27911 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240426 13:20:07.673687 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.673867 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1791) took 1432 ms. Trace:
I20240426 13:20:07.674458 27911 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:07.675024 27911 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.675725 27911 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.674376 25097 rpcz_store.cc:276] 0426 13:20:06.240962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.241183 (+ 221us) service_pool.cc:225] Handling call
0426 13:20:06.285283 (+ 44100us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:06.310969 (+ 25686us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:06.311004 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:06.311470 (+ 466us) log.cc:842] Serialized 26 byte log entry
0426 13:20:06.978294 (+666824us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:06.978331 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:06.978394 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:07.673566 (+695172us) raft_consensus.cc:1671] finished
0426 13:20:07.673601 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:07.673775 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1209,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":666458,"threads_started":1,"wal-append.queue_time_us":1324373}
I20240426 13:20:07.677500 25097 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:07.662461 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:07.701272 24954 raft_consensus.cc:1261] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:07.679459 27943 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:07.732674 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.697003 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:07.716565 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.012s sys 0.012s
W20240426 13:20:07.736732 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s user 0.012s sys 0.012s
I20240426 13:20:07.717504 24955 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.714323 24951 raft_consensus.cc:1261] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240426 13:20:07.727238 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.740939 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.670778 24997 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.000s sys 0.016s
I20240426 13:20:07.656800 27451 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:07.743714 24947 raft_consensus.cc:1261] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:20:07.744125 24947 pending_rounds.cc:76] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:07.744344 24947 pending_rounds.cc:97] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:20:07.744488 25105 raft_consensus.cc:1226] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240426 13:20:07.744642 24947 raft_consensus.cc:2866] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.740928 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.727869 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54262.1us
I20240426 13:20:07.753212 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.754299 25024 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:07.699465 27653 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:07.756086 24995 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.013s sys 0.024s
I20240426 13:20:07.756321 24954 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 2.
I20240426 13:20:07.707265 27916 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.737521 27869 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:07.737663 28044 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.015s sys 0.007s
W20240426 13:20:07.780344 24995 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.799s user 0.015s sys 0.024s
W20240426 13:20:07.784771 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 803799us
I20240426 13:20:07.786317 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.721949 25100 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:07.788681 25589 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:07.782707 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.669643 28061 consensus_queue.cc:1035] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:07.679816 27923 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:07.798236 27923 consensus_queue.cc:1225] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:07.773898 27964 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
W20240426 13:20:07.754717 27808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.016s sys 0.025s
I20240426 13:20:07.762125 28034 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:07.784380 28044 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.919s user 0.016s sys 0.008s
W20240426 13:20:07.799952 24995 leader_election.cc:334] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.683s (SENT)
I20240426 13:20:07.800499 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.815639 24954 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.815294 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:07.819723 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.774969 27451 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.847359 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.849807 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.851362 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51450us
W20240426 13:20:07.852324 24995 leader_election.cc:334] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.704s (SENT)
I20240426 13:20:07.853147 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:07.850668 24955 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 2.
W20240426 13:20:07.808441 27808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s user 0.016s sys 0.025s
I20240426 13:20:07.856859 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.859236 25100 raft_consensus.cc:1203] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-9.8] Dedup: 4.6->[7.7-9.8]
I20240426 13:20:07.859786 25100 pending_rounds.cc:76] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 6
I20240426 13:20:07.800618 27759 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:07.810966 27923 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:07.861678 24995 leader_election.cc:334] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.948s (SENT)
I20240426 13:20:07.872606 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:07.862474 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2015) took 1070 ms. Trace:
W20240426 13:20:07.858116 25029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.004s sys 0.018s
I20240426 13:20:07.859575 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.811798 28034 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:07.860256 25100 pending_rounds.cc:97] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20240426 13:20:07.801355 27916 raft_consensus.cc:2741] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20240426 13:20:07.875874 25029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s user 0.005s sys 0.018s
W20240426 13:20:07.854463 27630 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.029s sys 0.024s
I20240426 13:20:07.875582 24947 rpcz_store.cc:276] 0426 13:20:06.791958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.330685 (+538727us) service_pool.cc:225] Handling call
0426 13:20:07.330974 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.745262 (+414288us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.862213 (+116951us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54003c41d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:07.862410 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":948391296}
I20240426 13:20:07.876581 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.887492 24951 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:07.879860 25100 raft_consensus.cc:2866] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:07.786114 27869 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.857439 27498 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.009s sys 0.013s
I20240426 13:20:07.882656 26679 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:07.883320 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:07.854787 27653 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:07.895197 26679 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:07.872767 27810 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.003s sys 0.020s
I20240426 13:20:07.858228 27633 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:07.860487 27853 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.890219 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:07.860675 27438 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:07.870545 28036 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:07.918910 28036 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:07.919476 28036 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:07.892616 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:07.855110 27451 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:07.898017 27900 consensus_meta.cc:287] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.824s user 0.008s sys 0.001s
I20240426 13:20:07.898517 27943 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:07.903035 27647 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.903723 27755 consensus_queue.cc:1035] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:07.889292 27923 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:07.906210 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2101) took 1609 ms. Trace:
W20240426 13:20:07.942174 25022 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.011s sys 0.014s
W20240426 13:20:07.942803 25022 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.618s user 0.011s sys 0.014s
W20240426 13:20:07.944824 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 61593.2us
I20240426 13:20:07.888044 24995 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.917084 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:07.946516 25099 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:07.886587 27630 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.029s sys 0.025s
I20240426 13:20:07.870523 27759 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:07.920935 25101 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.007s sys 0.017s
W20240426 13:20:07.895814 27498 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.838s user 0.010s sys 0.013s
I20240426 13:20:07.923568 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.956377 25588 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.007s sys 0.018s
W20240426 13:20:07.906119 27810 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.791s user 0.003s sys 0.021s
I20240426 13:20:07.945844 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:07.947996 25096 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.010s sys 0.011s
W20240426 13:20:07.958058 25096 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.010s sys 0.012s
W20240426 13:20:07.947683 25026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.011s sys 0.010s
I20240426 13:20:07.935978 25029 rpcz_store.cc:276] 0426 13:20:06.277647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.277825 (+ 178us) service_pool.cc:225] Handling call
0426 13:20:06.278092 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:06.278263 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.140307 (+862044us) log.cc:842] Serialized 14 byte log entry
0426 13:20:07.886494 (+746187us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:07.886582 (+ 88us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:07.886637 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.886709 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:07.886817 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4701,"prepare.queue_time_us":1145741,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":16317,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":287,"thread_start_us":1563982,"threads_started":2}
I20240426 13:20:07.959403 26679 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:07.959800 26679 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:07.960130 26679 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:07.960156 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:07.960819 25027 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.011s sys 0.024s
W20240426 13:20:07.961145 25027 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.011s sys 0.024s
I20240426 13:20:07.961113 25029 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:07.955729 25101 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.743s user 0.007s sys 0.017s
I20240426 13:20:07.900900 28065 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:07.858027 28062 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:07.888681 28047 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:07.922279 27853 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.957520 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 69594.4us
I20240426 13:20:07.958792 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
W20240426 13:20:07.963768 25026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.012s sys 0.010s
I20240426 13:20:07.963716 24954 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:07.965159 28049 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.004s sys 0.020s
I20240426 13:20:07.930721 27900 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:07.957221 25588 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.622s user 0.008s sys 0.018s
I20240426 13:20:07.971490 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:07.981045 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:07.987910 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 663000us
I20240426 13:20:07.988514 24954 raft_consensus.cc:2354] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9.
I20240426 13:20:07.971534 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:07.981640 25103 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.005s sys 0.025s
I20240426 13:20:07.981030 24995 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:07.957738 27730 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.031s sys 0.023s
I20240426 13:20:07.983522 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:07.994292 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:07.956655 27759 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:07.979439 27725 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.001s sys 0.026s
W20240426 13:20:07.991094 25103 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.780s user 0.005s sys 0.025s
I20240426 13:20:08.002594 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.003032 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.003324 25624 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:08.004170 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.003677 25024 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:08.004761 25029 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.005633 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.006531 25069 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.007547 24956 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
W20240426 13:20:07.993223 24995 connection.cc:232] RPC call timeout handler was delayed by -1.0057191470000362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
W20240426 13:20:07.983667 25025 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.001s sys 0.026s
I20240426 13:20:07.972330 27943 raft_consensus.cc:2741] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20240426 13:20:07.984452 28049 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.993s user 0.005s sys 0.021s
I20240426 13:20:07.989742 27653 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240426 13:20:08.010089 25025 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.001s sys 0.026s
W20240426 13:20:08.010473 24995 connection.cc:232] RPC call timeout handler was delayed by -1.004513162999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20240426 13:20:08.025738 24995 leader_election.cc:334] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.988s (SENT)
I20240426 13:20:08.026350 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:08.011443 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.006s sys 0.023s
I20240426 13:20:08.016803 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:07.994226 27730 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.032s sys 0.024s
W20240426 13:20:08.021111 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.000s sys 0.034s
W20240426 13:20:08.006616 27725 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.002s sys 0.027s
I20240426 13:20:08.032407 27725 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:08.032446 28047 raft_consensus.cc:2726] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:08.032951 27725 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:08.033226 27799 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.009s sys 0.020s
I20240426 13:20:08.033448 27725 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:08.033584 27799 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.009s sys 0.020s
I20240426 13:20:08.034121 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1995) took 1726 ms. Trace:
I20240426 13:20:08.034513 24952 rpcz_store.cc:276] 0426 13:20:06.307506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.703077 (+395571us) service_pool.cc:225] Handling call
0426 13:20:06.719621 (+ 16544us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:06.755714 (+ 36093us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:06.755762 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:06.810987 (+ 55225us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.737232 (+926245us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:07.737286 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.737373 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.033904 (+296531us) raft_consensus.cc:1671] finished
0426 13:20:08.033924 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.034065 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2821,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":16631,"thread_start_us":901751,"threads_started":1,"wal-append.queue_time_us":1178090}
I20240426 13:20:07.968310 27911 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.036206 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.036935 24952 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.037696 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.037994 27911 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.038236 24952 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:08.038686 27911 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:20:08.039273 27911 raft_consensus.cc:686] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.024819 27923 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.000304 28065 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:08.028669 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.041201 25024 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:08.030356 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s user 0.006s sys 0.024s
W20240426 13:20:08.030565 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s user 0.000s sys 0.034s
I20240426 13:20:08.003628 27916 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:08.048899 26679 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.004925 27382 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.086551 26679 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.089031 27382 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.026993 28061 consensus_queue.cc:1035] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.069s
I20240426 13:20:08.090970 28061 consensus_queue.cc:1225] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240426 13:20:08.091759 27382 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.096320 24955 raft_consensus.cc:2396] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:08.097610 27759 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.051506 27916 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.101737 27847 raft_consensus.cc:3032] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:08.103464 27911 consensus_queue.cc:237] T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.106936 24947 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.026801 27653 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.100610 27916 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:07.981146 28062 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:08.133158 25023 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.022s sys 0.025s
W20240426 13:20:08.133605 25023 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s user 0.022s sys 0.025s
I20240426 13:20:08.122740 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.045435 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
W20240426 13:20:08.105947 27271 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.506s user 0.008s sys 0.000s
I20240426 13:20:08.135215 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.099792 27759 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.128177 25024 raft_consensus.cc:2454] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:08.109642 27847 raft_consensus.cc:729] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:08.122967 27677 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 1.059s user 0.011s sys 0.003s
W20240426 13:20:08.133381 27350 consensus_meta.cc:287] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 1.184s user 0.007s sys 0.006s
I20240426 13:20:08.144060 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.122766 27653 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:08.114660 27708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.020s sys 0.013s
W20240426 13:20:08.121307 27928 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.012s sys 0.016s
I20240426 13:20:08.148816 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.144800 24954 raft_consensus.cc:2396] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:08.154330 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 539) took 1323 ms. Trace:
W20240426 13:20:08.121474 27868 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.008s sys 0.019s
W20240426 13:20:08.133949 27707 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.883s user 0.005s sys 0.024s
W20240426 13:20:08.144806 25104 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.018s sys 0.029s
W20240426 13:20:08.185890 25104 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.876s user 0.018s sys 0.034s
I20240426 13:20:08.185971 24956 rpcz_store.cc:276] 0426 13:20:06.830497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.364730 (+534233us) service_pool.cc:225] Handling call
0426 13:20:07.400122 (+ 35392us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.008062 (+607940us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.154055 (+145993us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540038efd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.154260 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396514176}
I20240426 13:20:08.187283 25024 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 2.
I20240426 13:20:08.192857 25615 raft_consensus.cc:1261] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240426 13:20:08.135332 27869 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.198038 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.198844 24955 raft_consensus.cc:2362] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8. Re-sending same reply.
I20240426 13:20:08.190577 25028 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
W20240426 13:20:08.209765 25588 leader_election.cc:334] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.035s (SENT)
I20240426 13:20:08.229823 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 542) took 1343 ms. Trace:
W20240426 13:20:08.232920 24995 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.191s (SENT)
I20240426 13:20:08.233443 24995 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:08.232975 24952 rpcz_store.cc:276] 0426 13:20:06.886539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.156637 (+1270098us) service_pool.cc:225] Handling call
0426 13:20:08.191986 (+ 35349us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.229387 (+ 37401us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.229427 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.229495 (+ 68us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.229537 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.229617 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.229757 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.187400 27707 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.936s user 0.009s sys 0.024s
I20240426 13:20:08.237831 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:08.269209 24995 leader_election.cc:334] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.309s (SENT)
I20240426 13:20:08.235705 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.274015 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:08.269327 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 544) took 1341 ms. Trace:
I20240426 13:20:08.276290 24956 rpcz_store.cc:276] 0426 13:20:06.927551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.187262 (+1259711us) service_pool.cc:225] Handling call
0426 13:20:08.239138 (+ 51876us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.268999 (+ 29861us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.269023 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.269076 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.269104 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.269154 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.269273 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.269881 24948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.028s sys 0.026s
W20240426 13:20:08.277711 24948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.028s sys 0.026s
W20240426 13:20:08.230545 28025 consensus_meta.cc:287] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.963s user 0.015s sys 0.000s
W20240426 13:20:08.228551 27708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s user 0.024s sys 0.013s
I20240426 13:20:08.147320 28047 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:08.296017 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 1996) took 1988 ms. Trace:
I20240426 13:20:08.281723 24955 raft_consensus.cc:1226] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:08.284444 24952 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 4. Ops: []
I20240426 13:20:08.341935 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 546) took 1380 ms. Trace:
W20240426 13:20:08.185894 27868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.010s sys 0.021s
W20240426 13:20:08.228750 27928 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.012s sys 0.019s
I20240426 13:20:08.145316 27271 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.290823 24995 leader_election.cc:334] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.801s (SENT)
I20240426 13:20:08.327695 27900 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.202836 27451 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.321980 24951 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:08.328342 24954 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:08.329587 24956 raft_consensus.cc:1261] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240426 13:20:08.282716 27943 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:08.281195 27677 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:08.160288 27350 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.378500 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 550) took 1297 ms. Trace:
I20240426 13:20:08.383822 24956 rpcz_store.cc:276] 0426 13:20:07.080927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.277525 (+1196598us) service_pool.cc:225] Handling call
0426 13:20:08.320927 (+ 43402us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.378174 (+ 57247us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.378426 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.320981 25621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.013s sys 0.031s
I20240426 13:20:08.307255 28047 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.342409 24952 rpcz_store.cc:276] 0426 13:20:06.961123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.234640 (+1273517us) service_pool.cc:225] Handling call
0426 13:20:08.284253 (+ 49613us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.341610 (+ 57357us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.341852 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.343501 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 442331us
I20240426 13:20:08.389384 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 541) took 1541 ms. Trace:
W20240426 13:20:08.344249 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 744542us
W20240426 13:20:08.355008 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 349418us
I20240426 13:20:08.389807 24947 rpcz_store.cc:276] 0426 13:20:06.847579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.155745 (+1308166us) service_pool.cc:225] Handling call
0426 13:20:08.199551 (+ 43806us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.236987 (+ 37436us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.272256 (+ 35269us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.272578 (+ 322us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:08.272749 (+ 171us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.273035 (+ 286us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.389225 (+116190us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b54002cb0d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.389330 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99970944}
I20240426 13:20:08.391567 24947 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:08.355990 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 545) took 1413 ms. Trace:
I20240426 13:20:08.395787 24955 rpcz_store.cc:276] 0426 13:20:06.942446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.206118 (+1263672us) service_pool.cc:225] Handling call
0426 13:20:08.281534 (+ 75416us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.355704 (+ 74170us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.355931 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.340879 27708 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:08.400697 27708 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.404376 27708 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.414676 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 552) took 1333 ms. Trace:
I20240426 13:20:08.415414 24947 rpcz_store.cc:276] 0426 13:20:07.081485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.391009 (+1309524us) service_pool.cc:225] Handling call
0426 13:20:08.391352 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.414341 (+ 22989us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.414610 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.361438 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1804) took 1249 ms. Trace:
I20240426 13:20:08.268515 25024 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:08.269421 28065 consensus_queue.cc:1035] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.103s
I20240426 13:20:08.379885 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 551) took 1298 ms. Trace:
I20240426 13:20:08.341990 24948 rpcz_store.cc:276] 0426 13:20:06.307795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:06.705217 (+397422us) service_pool.cc:225] Handling call
0426 13:20:06.705617 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:06.705825 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.515340 (+809515us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.295550 (+780210us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.295626 (+ 76us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.295688 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.295789 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.295949 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":963587,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":19194,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9101,"thread_start_us":1557411,"threads_started":2}
I20240426 13:20:08.380743 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.424657 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.425364 24954 rpcz_store.cc:276] 0426 13:20:07.081215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.280572 (+1199357us) service_pool.cc:225] Handling call
0426 13:20:08.327059 (+ 46487us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.379554 (+ 52495us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.379821 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.428530 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1648) took 1346 ms. Trace:
I20240426 13:20:08.429056 24947 rpcz_store.cc:276] 0426 13:20:07.082031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.416599 (+1334568us) service_pool.cc:225] Handling call
0426 13:20:08.416974 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.428104 (+ 11130us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.428143 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.428199 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.428242 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.428322 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.428460 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.421496 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.413051 25589 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:08.424001 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 144434us
W20240426 13:20:08.432838 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 2.959s (SENT)
I20240426 13:20:08.434062 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:08.424101 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.436414 24952 raft_consensus.cc:2379] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.424916 25070 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.355626 27410 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.438519 24953 raft_consensus.cc:2354] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:08.437407 24950 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:08.399164 25621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.013s sys 0.032s
I20240426 13:20:08.387383 28047 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:08.442788 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.447594 24948 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:08.467981 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.468276 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1656) took 1372 ms. Trace:
I20240426 13:20:08.468626 24948 rpcz_store.cc:276] 0426 13:20:07.095776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.425583 (+1329807us) service_pool.cc:225] Handling call
0426 13:20:08.425897 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.467992 (+ 42095us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.468224 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.382699 27853 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.471253 24921 leader_election.cc:398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.471578 24921 leader_election.cc:401] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240426 13:20:08.442365 24947 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240426 13:20:08.409868 27708 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.451714 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:08.480990 24919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.017s sys 0.007s
W20240426 13:20:08.481460 24919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.830s user 0.017s sys 0.007s
W20240426 13:20:08.476480 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 469952us
I20240426 13:20:08.477356 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.288972 27847 consensus_queue.cc:260] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.477576 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1660) took 1215 ms. Trace:
I20240426 13:20:08.424633 27932 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:08.420182 27677 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.470778 24948 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240426 13:20:08.476750 24952 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240426 13:20:08.506124 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1662) took 1202 ms. Trace:
I20240426 13:20:08.506603 24952 rpcz_store.cc:276] 0426 13:20:07.305326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.441966 (+1136640us) service_pool.cc:225] Handling call
0426 13:20:08.476357 (+ 34391us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.505774 (+ 29417us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.506057 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.492604 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102359us
I20240426 13:20:08.487403 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:08.508781 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.418877 25101 rpcz_store.cc:276] 0426 13:20:07.111831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.212178 (+100347us) service_pool.cc:225] Handling call
0426 13:20:07.212456 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.212680 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:07.966572 (+753892us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.361001 (+394429us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.361076 (+ 75us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.361144 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.361231 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.361372 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":666908,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":622,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":13982,"thread_start_us":1143377,"threads_started":2}
I20240426 13:20:08.492551 24947 rpcz_store.cc:276] 0426 13:20:07.262279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.440253 (+1177974us) service_pool.cc:225] Handling call
0426 13:20:08.441038 (+ 785us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.475326 (+ 34288us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.477276 (+ 1950us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:08.503454 25099 consensus_meta.cc:287] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.556s user 0.009s sys 0.001s
I20240426 13:20:08.495278 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.510453 25099 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:08.505828 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2016) took 1107 ms. Trace:
I20240426 13:20:08.513671 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.514089 24948 rpcz_store.cc:276] 0426 13:20:07.398180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.469350 (+1071170us) service_pool.cc:225] Handling call
0426 13:20:08.470420 (+ 1070us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.505564 (+ 35144us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.505761 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1663104}
I20240426 13:20:08.513666 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51378 (request call id 1544) took 1110 ms. Trace:
I20240426 13:20:08.515377 25024 rpcz_store.cc:276] 0426 13:20:07.403400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.495010 (+1091610us) service_pool.cc:225] Handling call
0426 13:20:08.513605 (+ 18595us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.517657 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.470921 27869 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:08.518754 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:08.520399 24947 raft_consensus.cc:1261] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:08.523553 24948 raft_consensus.cc:1261] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20240426 13:20:08.523932 24948 pending_rounds.cc:76] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:08.524291 24948 pending_rounds.cc:97] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20240426 13:20:08.524518 24948 raft_consensus.cc:2866] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:08.525452 26679 consensus_queue.cc:1046] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:08.526464 26679 raft_consensus.cc:3032] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:08.526947 26679 raft_consensus.cc:729] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.527328 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.528498 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:08.529553 25024 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 7: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:08.466331 27410 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.527878 26679 consensus_queue.cc:260] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.531618 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.536199 24947 raft_consensus.cc:2362] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8. Re-sending same reply.
I20240426 13:20:08.535072 24956 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:08.536846 26679 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:08.506733 27900 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.537238 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1670) took 1009 ms. Trace:
I20240426 13:20:08.545094 24956 rpcz_store.cc:276] 0426 13:20:07.528118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.517018 (+988900us) service_pool.cc:225] Handling call
0426 13:20:08.534752 (+ 17734us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.536988 (+ 2236us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.537190 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:08.545094 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1668) took 1112 ms. Trace:
I20240426 13:20:08.546334 24951 rpcz_store.cc:276] 0426 13:20:07.432442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.894209 (+461767us) service_pool.cc:225] Handling call
0426 13:20:08.544895 (+650686us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000794d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.545038 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":838136064}
I20240426 13:20:08.522421 27869 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.542347 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:08.543556 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1801) took 1432 ms. Trace:
I20240426 13:20:08.517038 27498 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:08.517361 27847 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:08.522131 27928 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:08.522287 28097 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:08.541982 25029 consensus_meta.cc:287] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.519s user 0.007s sys 0.000s
I20240426 13:20:08.550772 25098 raft_consensus.cc:1203] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 6.4->[7.5-7.5] Dedup: 7.5->[]
I20240426 13:20:08.563195 25029 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:08.551410 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1807) took 1431 ms. Trace:
I20240426 13:20:08.521629 28062 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.516280 27932 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:08.568488 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 47782.9us
I20240426 13:20:08.565306 25098 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 9. Ops: [7.5-7.5]
W20240426 13:20:08.554855 27946 consensus_meta.cc:287] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.095s user 0.010s sys 0.002s
I20240426 13:20:08.552762 25103 rpcz_store.cc:276] 0426 13:20:07.111122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.210798 (+ 99676us) service_pool.cc:225] Handling call
0426 13:20:07.211108 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.211290 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.002495 (+791205us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.543195 (+540700us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.543267 (+ 72us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.543321 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.543382 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.543491 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49682,"prepare.queue_time_us":638961,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":69956,"thread_start_us":1259582,"threads_started":2}
I20240426 13:20:08.569453 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.520181 27677 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.549708 28035 raft_consensus.cc:2741] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:08.604671 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.605173 27759 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:08.605772 27759 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:08.605986 28074 log.cc:924] Time spent T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240426 13:20:08.606158 27759 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.606750 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:08.554929 27869 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:08.608120 28082 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20240426 13:20:08.608345 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.608704 24956 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20240426 13:20:08.609028 24947 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:08.609776 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.609944 27900 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:08.610306 27900 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.610689 27900 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:08.610893 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.612214 27647 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:08.609779 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.612705 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2103) took 1411 ms. Trace:
I20240426 13:20:08.613013 27647 raft_consensus.cc:686] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:08.613073 25030 rpcz_store.cc:276] 0426 13:20:07.201761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.201879 (+ 118us) service_pool.cc:225] Handling call
0426 13:20:07.267797 (+ 65918us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:07.288646 (+ 20849us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.288675 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:07.295989 (+ 7314us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.043323 (+747334us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.043365 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.043433 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.612496 (+569063us) raft_consensus.cc:1671] finished
0426 13:20:08.612514 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.612652 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22,"raft.queue_time_us":20659,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":7104,"thread_start_us":722335,"threads_started":1,"wal-append.queue_time_us":1166744}
I20240426 13:20:08.614053 27647 consensus_queue.cc:237] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.615609 27738 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:08.594489 25070 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.557365 27498 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:08.646382 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:08.603366 28087 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20240426 13:20:08.603499 25103 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:08.613111 24947 raft_consensus.cc:2362] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7. Re-sending same reply.
W20240426 13:20:08.616377 28080 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20240426 13:20:08.604734 27677 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.564497 27928 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:08.562813 28088 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20240426 13:20:08.617332 27271 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.562809 27451 raft_consensus.cc:2741] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:08.629957 27946 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.612337 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.570587 28097 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.644233 28085 log.cc:924] Time spent T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20240426 13:20:08.644081 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1534) took 1606 ms. Trace:
I20240426 13:20:08.675109 27932 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.645320 25028 raft_consensus.cc:2396] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:08.705185 27498 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:08.595947 28062 consensus_queue.cc:1035] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20240426 13:20:08.645870 24921 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.779953 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:08.780411 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 135272us
I20240426 13:20:08.781248 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:08.748704 27738 raft_consensus.cc:686] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
W20240426 13:20:08.605104 28093 log.cc:924] Time spent T d3a669ffe77a4ad29604f4ecd7c2644a P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240426 13:20:08.611907 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:08.645349 27725 consensus_meta.cc:287] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.611s user 0.006s sys 0.004s
I20240426 13:20:08.671590 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.694924 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 607) took 1251 ms. Trace:
I20240426 13:20:08.695614 27900 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.694196 27869 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.675613 28025 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.675240 27759 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.698240 27410 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.816419 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 553) took 1013 ms. Trace:
I20240426 13:20:08.816659 27869 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.802939 27451 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:08.817338 28025 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.817756 27451 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.818794 24921 leader_election.cc:398] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Vote denied by peer 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089) with higher term. Message: Invalid argument: T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9.
I20240426 13:20:08.819371 28025 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.819653 27451 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:08.731750 24995 leader_election.cc:334] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 3.000s (SENT)
I20240426 13:20:08.779939 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:08.798842 27911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.006s sys 0.004s
I20240426 13:20:08.598088 28055 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:08.832208 28055 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.833556 28097 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.834275 28097 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:08.834705 28097 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:08.834858 28055 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.835237 28097 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:08.751955 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.826471 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51378 (request call id 1541) took 1773 ms. Trace:
I20240426 13:20:08.836030 25622 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:08.837626 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1809) took 1444 ms. Trace:
I20240426 13:20:08.836792 25026 rpcz_store.cc:276] 0426 13:20:07.053534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.137779 (+ 84245us) service_pool.cc:225] Handling call
0426 13:20:07.159409 (+ 21630us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.159622 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.159645 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.198420 (+ 38775us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.981448 (+783028us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:07.981486 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.981554 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.826295 (+844741us) raft_consensus.cc:1671] finished
0426 13:20:08.826313 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.826413 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":675838,"threads_started":1,"wal-append.queue_time_us":1034774}
I20240426 13:20:08.838115 25096 rpcz_store.cc:276] 0426 13:20:07.392851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.392979 (+ 128us) service_pool.cc:225] Handling call
0426 13:20:07.413332 (+ 20353us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.414038 (+ 706us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.414871 (+ 833us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.420859 (+ 5988us) log.cc:842] Serialized 26 byte log entry
0426 13:20:07.958534 (+537675us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:07.964305 (+ 5771us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.964410 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.837354 (+872944us) raft_consensus.cc:1671] finished
0426 13:20:08.837382 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.837556 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":534440,"threads_started":1,"wal-append.queue_time_us":909954}
W20240426 13:20:08.677423 27916 consensus_meta.cc:287] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.535s user 0.004s sys 0.004s
I20240426 13:20:08.756110 24948 raft_consensus.cc:2379] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.843333 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:08.674312 28034 consensus_meta.cc:287] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.785s user 0.009s sys 0.000s
I20240426 13:20:08.844462 28034 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.677989 28036 consensus_meta.cc:287] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.757s user 0.009s sys 0.000s
I20240426 13:20:08.769335 25023 rpcz_store.cc:276] 0426 13:20:07.037677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.128961 (+ 91284us) service_pool.cc:225] Handling call
0426 13:20:07.129245 (+ 284us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.130669 (+ 1424us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.130697 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.131194 (+ 497us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.140335 (+1009141us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:08.141114 (+ 779us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.150355 (+ 9241us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.150432 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.616484 (+466052us) raft_consensus.cc:1671] finished
0426 13:20:08.616503 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.644005 (+ 27502us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8966,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":10028,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6508,"thread_start_us":1002497,"threads_started":1,"wal-append.queue_time_us":1293510}
W20240426 13:20:08.644692 28076 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
I20240426 13:20:08.647264 24951 raft_consensus.cc:1261] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:08.850721 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 294479us
I20240426 13:20:08.860472 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:08.865705 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:08.753383 26679 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:08.933547 28106 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240426 13:20:08.933441 26679 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.934362 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.935559 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.935922 26679 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.936435 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:08.777516 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58866 (request call id 1524) took 1349 ms. Trace:
I20240426 13:20:08.570272 28061 consensus_queue.cc:1035] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.071s
I20240426 13:20:08.752765 27928 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:08.754278 27868 raft_consensus.cc:2781] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:08.937161 28061 consensus_queue.cc:1225] T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer 1e87e7bb414642aeb72fe0d2ff72cc2e log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240426 13:20:08.937506 27868 raft_consensus.cc:686] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:08.938437 27868 consensus_queue.cc:237] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.936928 25098 rpcz_store.cc:276] 0426 13:20:07.427768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.442013 (+ 14245us) service_pool.cc:225] Handling call
0426 13:20:08.550665 (+1108652us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.569185 (+ 18520us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.777316 (+208131us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540009d8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.777459 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2548854656}
I20240426 13:20:08.943609 25098 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240426 13:20:08.778142 25618 raft_consensus.cc:3037] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:08.703907 28047 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.802439 27725 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.944885 27928 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.945513 27725 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.945701 25029 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
W20240426 13:20:08.809989 28092 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20240426 13:20:08.798049 25621 rpcz_store.cc:276] 0426 13:20:07.443955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.449746 (+ 5791us) service_pool.cc:225] Handling call
0426 13:20:07.496532 (+ 46786us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.550794 (+ 54262us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.556868 (+ 6074us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.663402 (+106534us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.479157 (+815755us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:08.491799 (+ 12642us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.505584 (+ 13785us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.694644 (+189060us) raft_consensus.cc:1671] finished
0426 13:20:08.694681 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.694855 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4265,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13997,"thread_start_us":672875,"threads_started":1,"wal-append.queue_time_us":961555}
I20240426 13:20:08.812314 28096 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.947113 27928 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.947043 28096 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:08.947662 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2115) took 1727 ms. Trace:
I20240426 13:20:08.813140 27759 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.814697 27900 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.820421 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2111) took 1607 ms. Trace:
I20240426 13:20:08.699030 27350 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:08.953923 25031 rpcz_store.cc:276] 0426 13:20:07.213366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.307744 (+ 94378us) service_pool.cc:225] Handling call
0426 13:20:07.409840 (+102096us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.415672 (+ 5832us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.415714 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:07.489427 (+ 73713us) log.cc:842] Serialized 548 byte log entry
0426 13:20:08.043788 (+554361us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:08.043829 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.043895 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.820106 (+776211us) raft_consensus.cc:1671] finished
0426 13:20:08.820134 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.820346 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":553964,"threads_started":1,"wal-append.queue_time_us":1323511}
W20240426 13:20:08.820693 28101 log.cc:924] Time spent T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20240426 13:20:08.821867 24956 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.824532 24995 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:08.740514 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:08.957923 24947 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:08.784188 27923 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:08.978600 27923 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:08.979192 27498 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.742115 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 147586us
I20240426 13:20:08.830003 27738 consensus_queue.cc:237] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.008929 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1812) took 1104 ms. Trace:
I20240426 13:20:08.819662 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:08.772282 27799 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20240426 13:20:08.847746 24948 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:08.866639 25096 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:08.868319 27451 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:08.921660 28034 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:08.922401 27946 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.015708 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.922905 25103 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
W20240426 13:20:09.016026 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 284300us
I20240426 13:20:08.923024 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2108) took 1718 ms. Trace:
W20240426 13:20:09.017161 24995 leader_election.cc:334] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 3.322s (SENT)
W20240426 13:20:08.923800 28072 log.cc:924] Time spent T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.369s user 0.000s sys 0.001s
I20240426 13:20:08.859120 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.023677 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.024369 25619 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.025167 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.859249 28036 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:09.027133 28036 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.027707 25097 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:09.028347 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1647) took 1946 ms. Trace:
I20240426 13:20:09.028807 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.016978 25022 rpcz_store.cc:276] 0426 13:20:07.204149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.252128 (+ 47979us) service_pool.cc:225] Handling call
0426 13:20:07.273648 (+ 21520us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:07.306258 (+ 32610us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:20:07.943347 (+637089us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.353849 (+410502us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:08.354497 (+ 648us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.354673 (+ 176us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:08.354717 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.354803 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.922793 (+567990us) raft_consensus.cc:1671] finished
0426 13:20:08.922816 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.922969 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18770,"prepare.queue_time_us":350229,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":6433,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":4962,"thread_start_us":980447,"threads_started":2,"wal-append.queue_time_us":816229}
I20240426 13:20:09.029335 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1659) took 1834 ms. Trace:
I20240426 13:20:09.017093 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:08.850358 27911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s user 0.006s sys 0.005s
I20240426 13:20:08.931157 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.932268 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:08.911036 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 48152us
I20240426 13:20:08.934453 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.948011 25029 rpcz_store.cc:276] 0426 13:20:07.219775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.563987 (+1344212us) service_pool.cc:225] Handling call
0426 13:20:08.564306 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.946114 (+381808us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.947435 (+ 1321us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400242f54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.947611 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876686208}
I20240426 13:20:09.033718 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.034310 25029 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:08.867576 27916 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:08.699041 27653 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.000s
I20240426 13:20:08.945513 28047 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:08.955147 28107 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240426 13:20:08.955787 28055 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
W20240426 13:20:08.963940 27964 consensus_meta.cc:287] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 1.163s user 0.008s sys 0.007s
I20240426 13:20:08.968315 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:08.968667 25621 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:08.968924 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1665) took 1587 ms. Trace:
I20240426 13:20:08.970642 28096 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:08.564378 25104 rpcz_store.cc:276] 0426 13:20:07.119506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.257270 (+137764us) service_pool.cc:225] Handling call
0426 13:20:07.290331 (+ 33061us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.290562 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.186424 (+895862us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.550386 (+363962us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.550440 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.550486 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.550554 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.551353 (+ 799us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36809,"prepare.queue_time_us":618122,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":36971,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3290,"thread_start_us":1202827,"threads_started":2}
I20240426 13:20:08.971565 27410 consensus_queue.cc:1046] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240426 13:20:08.974606 24950 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 547) took 1938 ms (client timeout 1752 ms). Trace:
I20240426 13:20:08.974745 24949 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 7. Ops: []
I20240426 13:20:08.975834 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:08.976013 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2105) took 1773 ms. Trace:
W20240426 13:20:08.977911 28086 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.506s user 0.001s sys 0.001s
I20240426 13:20:08.817001 25100 rpcz_store.cc:276] 0426 13:20:07.803092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.810662 (+ 7570us) service_pool.cc:225] Handling call
0426 13:20:07.811316 (+ 654us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:07.937025 (+125709us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:07.937060 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:07.937596 (+ 536us) log.cc:842] Serialized 52 byte log entry
0426 13:20:08.474629 (+537033us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.474711 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.474815 (+ 104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.816109 (+341294us) raft_consensus.cc:1671] finished
0426 13:20:08.816143 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.816349 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":465506,"threads_started":1,"wal-append.queue_time_us":862723}
I20240426 13:20:09.031023 27647 raft_consensus.cc:2726] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:08.840224 28025 raft_consensus.cc:2781] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
W20240426 13:20:08.981199 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 471100us
I20240426 13:20:09.088948 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1810) took 1657 ms. Trace:
I20240426 13:20:08.994880 25101 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:09.013868 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:20:09.090543 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 76757us
I20240426 13:20:09.009361 25105 rpcz_store.cc:276] 0426 13:20:07.904270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.906652 (+ 2382us) service_pool.cc:225] Handling call
0426 13:20:07.921920 (+ 15268us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.156749 (+234829us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.156777 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.225429 (+ 68652us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.500556 (+275127us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:08.500591 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.994082 (+493491us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54000ea954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:08.994136 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.994179 (+ 43us) raft_consensus.cc:1671] finished
0426 13:20:08.994192 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.008859 (+ 14667us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":428,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":21785,"spinlock_wait_cycles":539274624,"thread_start_us":263585,"threads_started":1,"wal-append.queue_time_us":631649}
I20240426 13:20:08.923632 27869 consensus_queue.cc:1035] T 6324309bd7cf40778c508e740dcf979d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:09.022425 27350 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.022857 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.023048 28049 raft_consensus.cc:2726] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.028810 24955 rpcz_store.cc:276] 0426 13:20:07.081764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.396708 (+1314944us) service_pool.cc:225] Handling call
0426 13:20:08.446747 (+ 50039us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.473828 (+ 27081us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.473863 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.474306 (+ 443us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.705804 (+231498us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:08.706177 (+ 373us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.706266 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.028146 (+321880us) raft_consensus.cc:1671] finished
0426 13:20:09.028166 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.028294 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4550,"raft.queue_time_us":3685,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":12840,"thread_start_us":174349,"threads_started":1,"wal-append.queue_time_us":516380}
I20240426 13:20:09.105388 24995 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.031463 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:09.045892 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 226297us
I20240426 13:20:09.107146 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.046710 24947 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:08.973235 23796 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240426 13:20:09.048043 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.048786 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.108683 24956 raft_consensus.cc:2379] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.051091 26679 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:09.050786 24953 rpcz_store.cc:276] 0426 13:20:07.381891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.659328 (+277437us) service_pool.cc:225] Handling call
0426 13:20:08.968662 (+1309334us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54002cbfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.968830 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1556543872}
I20240426 13:20:09.051910 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.052251 25069 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.016441 27451 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:09.058110 24950 rpcz_store.cc:269] 0426 13:20:07.036069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.117384 (+ 81315us) service_pool.cc:225] Handling call
0426 13:20:08.974351 (+1856967us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000785d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:08.974499 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2977547520}
I20240426 13:20:09.105865 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.116714 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.118031 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.118295 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.118700 24956 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:09.119457 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.058566 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.065244 25029 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:08.980765 27923 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:09.070497 28075 log.cc:924] Time spent T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.782s user 0.003s sys 0.001s
I20240426 13:20:09.057744 25025 rpcz_store.cc:276] 0426 13:20:07.202327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.202444 (+ 117us) service_pool.cc:225] Handling call
0426 13:20:07.202699 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:07.202891 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:08.023819 (+820928us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.443584 (+419765us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.822527 (+378943us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ba7c6 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351
0426 13:20:08.830079 (+ 7552us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:08.975549 (+145470us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b18000ae8e0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d365faa 00007fda96a54f25 00007fda8d3ea094 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8 00007fda90332168
0426 13:20:08.975687 (+ 138us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:08.975735 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:08.975778 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.975851 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:08.975952 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":527827,"prepare.queue_time_us":408596,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":558381,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":12645,"spinlock_wait_cycles":109254400,"thread_start_us":1130850,"threads_started":2,"wal-append.queue_time_us":542067}
I20240426 13:20:09.053064 27808 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240426 13:20:09.053170 27853 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.053313 27847 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20240426 13:20:09.059558 27911 consensus_queue.cc:1035] T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20240426 13:20:09.084499 27833 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:09.053539 28036 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20240426 13:20:09.086171 28094 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.620s user 0.001s sys 0.001s
I20240426 13:20:09.056933 28047 consensus_queue.cc:1035] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.030s
I20240426 13:20:09.087126 28062 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240426 13:20:09.173575 28062 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:09.174039 28062 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.046343 27271 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.142397 27810 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:09.089854 25102 rpcz_store.cc:276] 0426 13:20:07.431365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.450079 (+ 18714us) service_pool.cc:225] Handling call
0426 13:20:07.493172 (+ 43093us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:07.496531 (+ 3359us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:08.011173 (+514642us) log.cc:842] Serialized 14 byte log entry
0426 13:20:08.495143 (+483970us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.509839 (+ 14696us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.509985 (+ 146us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:08.510016 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:08.510077 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.088754 (+578677us) raft_consensus.cc:1671] finished
0426 13:20:09.088776 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.088897 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19349,"prepare.queue_time_us":412460,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":31972,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":9772,"thread_start_us":969961,"threads_started":2,"wal-append.queue_time_us":731740}
I20240426 13:20:09.091630 25619 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:09.092159 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1666) took 1693 ms. Trace:
I20240426 13:20:09.110201 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.139586 28131 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.115134 25024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2113) took 1901 ms. Trace:
I20240426 13:20:09.120899 24956 raft_consensus.cc:2379] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.029757 24954 rpcz_store.cc:276] 0426 13:20:07.194952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.427993 (+1233041us) service_pool.cc:225] Handling call
0426 13:20:08.470123 (+ 42130us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.504125 (+ 34002us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:08.662896 (+158771us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.008669 (+345773us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.029021 (+ 20352us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.029070 (+ 49us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.029118 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.029191 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.029288 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20937,"prepare.queue_time_us":330279,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":47917,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":366940,"threads_started":2}
W20240426 13:20:09.122795 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 70611.3us
I20240426 13:20:09.187944 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.190251 24956 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:09.190619 24956 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:09.124353 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.193559 25619 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20240426 13:20:09.130429 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 540) took 2294 ms. Trace:
I20240426 13:20:09.196195 24949 rpcz_store.cc:276] 0426 13:20:06.836118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.365400 (+529282us) service_pool.cc:225] Handling call
0426 13:20:08.974496 (+1609096us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.057896 (+ 83400us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.130255 (+ 72359us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54000785d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.130374 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3590358912}
I20240426 13:20:09.197646 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.198428 24949 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.199388 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.133270 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.130093 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.121713 25070 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:09.129671 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.131733 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.201414 25029 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:09.126551 24995 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.135445 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.139783 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.141568 24996 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:09.143142 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.187319 25024 rpcz_store.cc:276] 0426 13:20:07.213868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.560353 (+1346485us) service_pool.cc:225] Handling call
0426 13:20:08.560663 (+ 310us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:08.982384 (+421721us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.982420 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:08.982950 (+ 530us) log.cc:842] Serialized 52 byte log entry
0426 13:20:09.087277 (+104327us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b18000b1fa0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e291 00007fda902eb75f
0426 13:20:09.112597 (+ 25320us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:20:09.112649 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.112746 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.114807 (+ 2061us) raft_consensus.cc:1671] finished
0426 13:20:09.114824 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.115067 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":416418,"raft.queue_time_us":417251,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":17132,"spinlock_wait_cycles":11081984,"thread_start_us":25069,"threads_started":1,"wal-append.queue_time_us":116103}
I20240426 13:20:09.206529 24950 raft_consensus.cc:2396] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:09.207271 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.144162 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 557) took 1022 ms. Trace:
I20240426 13:20:09.207731 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.144536 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1815) took 1239 ms. Trace:
I20240426 13:20:09.047264 27653 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.208189 25101 rpcz_store.cc:276] 0426 13:20:08.121567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.554016 (+432449us) service_pool.cc:225] Handling call
0426 13:20:08.994698 (+440682us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.089999 (+ 95301us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.143872 (+ 53873us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54000ea950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.144091 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012761472}
I20240426 13:20:09.210789 27653 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.105420 28055 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.123191 27451 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:09.150631 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.103464 28065 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.211619 28055 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.212342 28065 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.214219 28055 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:09.103289 28025 raft_consensus.cc:686] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:09.215437 28025 consensus_queue.cc:237] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.215956 25099 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.205750 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.095283 27637 consensus_queue.cc:1035] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20240426 13:20:09.218412 27451 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.162717 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 405) took 1705 ms. Trace:
I20240426 13:20:09.162778 27498 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.219695 25100 raft_consensus.cc:1226] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
W20240426 13:20:09.172597 28132 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240426 13:20:09.219841 25097 rpcz_store.cc:276] 0426 13:20:07.905321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.029857 (+124536us) service_pool.cc:225] Handling call
0426 13:20:08.101460 (+ 71603us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.028093 (+926633us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.144241 (+116148us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b540001aed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:09.144479 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111356672}
I20240426 13:20:09.120024 27755 consensus_queue.cc:1035] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20240426 13:20:09.220683 27451 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:09.112172 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 137904us
I20240426 13:20:09.184201 24952 rpcz_store.cc:276] 0426 13:20:07.400881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.507576 (+1106695us) service_pool.cc:225] Handling call
0426 13:20:08.529476 (+ 21900us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.543975 (+ 14499us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.544006 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.545411 (+ 1405us) log.cc:842] Serialized 26 byte log entry
0426 13:20:08.799176 (+253765us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.091442 (+292266us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.091770 (+ 328us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.091861 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.091917 (+ 56us) raft_consensus.cc:1671] finished
0426 13:20:09.091934 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.092096 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19064,"prepare.queue_time_us":30583,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":17118,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":32011,"thread_start_us":461427,"threads_started":2,"wal-append.queue_time_us":431200}
I20240426 13:20:09.223783 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.219578 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a69cdcf2ee4768a20888dd974c6ba7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.224576 24952 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:09.224964 24953 raft_consensus.cc:3037] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.206414 24955 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 4.
I20240426 13:20:09.097632 27410 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:09.226635 27410 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:09.227064 27410 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.227138 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.224140 25024 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:09.191416 24956 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.228036 24955 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.208642 24950 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:09.232961 25029 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.214320 28065 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:09.154146 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.235317 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 81281.4us
I20240426 13:20:09.132261 28104 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.228510 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.236624 28126 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.236048 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.237486 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.238101 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.238237 25022 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.238425 24921 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.238813 25029 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.239288 27900 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.239874 27900 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:09.219635 25028 rpcz_store.cc:276] 0426 13:20:07.457704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.967753 (+1510049us) service_pool.cc:225] Handling call
0426 13:20:08.971011 (+ 3258us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.031318 (+ 60307us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.031349 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.038843 (+ 7494us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.114325 (+ 75482us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:09.114664 (+ 339us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.162413 (+ 47749us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.162485 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.162525 (+ 40us) raft_consensus.cc:1671] finished
0426 13:20:09.162539 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.162661 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94692,"prepare.queue_time_us":329,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":88621,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":22164,"thread_start_us":58356,"threads_started":2,"wal-append.queue_time_us":41467}
I20240426 13:20:09.225788 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.121693 27350 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.156333 28096 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.247282 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:09.247974 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 404) took 1792 ms. Trace:
I20240426 13:20:09.246482 25069 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.247231 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.244515 24947 raft_consensus.cc:2396] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:09.229274 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.249735 24954 raft_consensus.cc:1261] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20240426 13:20:09.250376 24954 pending_rounds.cc:76] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 4
I20240426 13:20:09.228829 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.251003 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 407) took 1768 ms. Trace:
I20240426 13:20:09.229758 24956 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.251505 25028 rpcz_store.cc:276] 0426 13:20:07.483093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.242091 (+1758998us) service_pool.cc:225] Handling call
0426 13:20:09.250551 (+ 8460us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.250720 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.250747 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.250787 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:09.250821 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.250869 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.250955 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.144264 27677 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240426 13:20:09.253949 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:09.249068 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 127431us
I20240426 13:20:09.257444 24955 raft_consensus.cc:2454] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:09.257499 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:09.256973 24953 raft_consensus.cc:2454] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.243377 24952 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.229090 24949 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.259629 24954 pending_rounds.cc:97] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240426 13:20:09.260747 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.259007 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.264954 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.265506 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.265690 24948 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.268898 24955 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:09.273320 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.274860 25069 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.277673 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.260725 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.285308 24954 raft_consensus.cc:2866] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.287699 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.288357 25024 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:09.249631 25031 rpcz_store.cc:276] 0426 13:20:07.455773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.956812 (+1501039us) service_pool.cc:225] Handling call
0426 13:20:08.957229 (+ 417us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.008808 (+ 51579us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.008851 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.009359 (+ 508us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.114510 (+105151us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.114822 (+ 312us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.247644 (+132822us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.247749 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.247790 (+ 41us) raft_consensus.cc:1671] finished
0426 13:20:09.247805 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.247924 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77098,"prepare.queue_time_us":416,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":80212,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":25732,"thread_start_us":162101,"threads_started":2,"wal-append.queue_time_us":75906}
I20240426 13:20:09.297148 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2123) took 1630 ms. Trace:
I20240426 13:20:09.297619 25031 rpcz_store.cc:276] 0426 13:20:07.667090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.296103 (+1629013us) service_pool.cc:225] Handling call
0426 13:20:09.296495 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.296724 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.296762 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.296820 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.296865 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.296933 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.297082 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.299332 25031 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:09.143994 28124 consensus_queue.cc:1046] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20240426 13:20:09.301182 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 409) took 1812 ms. Trace:
I20240426 13:20:09.145606 27853 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.308073 24947 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.307621 25028 rpcz_store.cc:276] 0426 13:20:07.488868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.253521 (+1764653us) service_pool.cc:225] Handling call
0426 13:20:09.300752 (+ 47231us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.300909 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.300931 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.300967 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.300996 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.301040 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.301136 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.303259 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.144917 28099 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.116s
I20240426 13:20:09.313994 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.312732 25022 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.310175 25070 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:09.314523 24956 raft_consensus.cc:1261] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240426 13:20:09.309947 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 410) took 1409 ms. Trace:
I20240426 13:20:09.320466 24956 pending_rounds.cc:76] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:09.320612 25028 rpcz_store.cc:276] 0426 13:20:07.900864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.309154 (+1408290us) service_pool.cc:225] Handling call
0426 13:20:09.309408 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.309555 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.309595 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.309648 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.309697 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.309763 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.309872 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.322489 25028 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:09.247020 27810 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:09.316190 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.313436 25031 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2126) took 1402 ms. Trace:
I20240426 13:20:09.323184 24952 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:09.323132 25031 rpcz_store.cc:276] 0426 13:20:07.897792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.298737 (+1400945us) service_pool.cc:225] Handling call
0426 13:20:09.299114 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.299789 (+ 675us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.299984 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.323596 24955 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:09.325037 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:09.320930 24956 pending_rounds.cc:97] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240426 13:20:09.325301 25069 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:09.325819 24956 raft_consensus.cc:2866] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.327190 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.328008 24956 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 8: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:09.253657 28126 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.333420 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.334024 24954 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:09.261003 28061 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:09.341349 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.341949 24953 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.315747 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:09.332396 25022 raft_consensus.cc:1226] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 8. Ops: []
I20240426 13:20:09.166067 28120 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.149s
I20240426 13:20:09.261389 27451 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.329502 24955 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.241709 27271 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.342605 24949 raft_consensus.cc:3037] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:09.284415 28104 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:09.314011 28086 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20240426 13:20:09.343454 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 121073us
I20240426 13:20:09.302946 27653 raft_consensus.cc:2781] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
W20240426 13:20:09.354387 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44313.5us
I20240426 13:20:09.356406 25618 raft_consensus.cc:1261] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240426 13:20:09.314607 27633 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
W20240426 13:20:09.359565 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 97170.2us
I20240426 13:20:09.360565 24921 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.315701 28047 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:09.320144 27708 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:09.175083 28062 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:09.370056 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 228585us
W20240426 13:20:09.401167 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 274706us
I20240426 13:20:09.284704 27923 consensus_queue.cc:1035] T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.064s
I20240426 13:20:09.405478 24950 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:09.405898 25022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:51352 (request call id 954) took 1663 ms. Trace:
I20240426 13:20:09.406234 25022 rpcz_store.cc:276] 0426 13:20:07.742669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.358206 (+1615537us) service_pool.cc:225] Handling call
0426 13:20:09.405831 (+ 47625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:09.406968 25024 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:09.403285 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.353379 24947 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.344930 27916 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:09.351248 27900 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240426 13:20:09.425791 28134 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240426 13:20:09.426548 24950 raft_consensus.cc:1261] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20240426 13:20:09.427281 28135 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240426 13:20:09.428551 27810 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.431385 27350 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.432114 28124 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:09.434935 24921 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.435451 24948 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:09.416728 24949 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:09.436796 24948 pending_rounds.cc:76] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 2
I20240426 13:20:09.437673 24948 pending_rounds.cc:97] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240426 13:20:09.437974 24948 raft_consensus.cc:2866] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.419317 24995 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.419494 26679 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:09.353271 28104 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.354575 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:09.418043 27755 consensus_queue.cc:1035] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.003s
I20240426 13:20:09.420822 28061 raft_consensus.cc:686] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.417198 24953 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:09.419330 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 94137.1us
I20240426 13:20:09.449787 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:09.364781 27725 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:09.428503 25023 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2118) took 2074 ms. Trace:
I20240426 13:20:09.453219 25023 rpcz_store.cc:276] 0426 13:20:07.354211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.900393 (+1546182us) service_pool.cc:225] Handling call
0426 13:20:08.919616 (+ 19223us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.997930 (+ 78314us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.997977 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.018413 (+ 20436us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.152273 (+133860us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:09.152314 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.152379 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:09.428337 (+275958us) raft_consensus.cc:1671] finished
0426 13:20:09.428353 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.428445 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77830,"raft.queue_time_us":77936,"spinlock_wait_cycles":6890240,"thread_start_us":93516,"threads_started":1,"wal-append.queue_time_us":212098}
I20240426 13:20:09.444492 25070 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:09.419044 28100 log.cc:924] Time spent T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240426 13:20:09.410248 27633 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:09.446666 28097 consensus_meta.cc:287] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.611s user 0.001s sys 0.009s
I20240426 13:20:09.452155 24953 pending_rounds.cc:76] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting all ops after (but not including) 3
I20240426 13:20:09.458492 24953 pending_rounds.cc:97] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20240426 13:20:09.458806 24953 raft_consensus.cc:2866] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:09.459890 26679 raft_consensus.cc:686] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.460956 26679 consensus_queue.cc:237] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.462568 24954 raft_consensus.cc:2454] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:09.458240 28126 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:09.465404 28061 consensus_queue.cc:237] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.466549 25027 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2109) took 2262 ms. Trace:
I20240426 13:20:09.466890 25027 rpcz_store.cc:276] 0426 13:20:07.204406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:07.296860 (+ 92454us) service_pool.cc:225] Handling call
0426 13:20:07.297187 (+ 327us) raft_consensus.cc:1457] Updating replica for 3 ops
0426 13:20:07.307586 (+ 10399us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:07.307619 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0426 13:20:07.322884 (+ 15265us) log.cc:842] Serialized 78 byte log entry
0426 13:20:07.961777 (+638893us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:07.981729 (+ 19952us) log.cc:842] Serialized 14 byte log entry
0426 13:20:07.985103 (+ 3374us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:07.985180 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:08.977316 (+992136us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:09.458739 (+481423us) raft_consensus.cc:1671] finished
0426 13:20:09.466334 (+ 7595us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.466475 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23557,"prepare.queue_time_us":19772,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":8145,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":21896,"spinlock_wait_cycles":59144448,"thread_start_us":597186,"threads_started":1,"wal-append.queue_time_us":1004760}
I20240426 13:20:09.468421 25029 raft_consensus.cc:1203] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 5.7->[7.8-9.9] Dedup: 9.9->[]
I20240426 13:20:09.468533 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2117) took 2216 ms. Trace:
I20240426 13:20:09.438482 24952 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:09.475206 28097 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.477821 28097 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.447517 28062 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:09.488479 25029 raft_consensus.cc:1226] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
W20240426 13:20:09.490801 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46413.2us
I20240426 13:20:09.492384 25070 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:09.443195 28104 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:09.493652 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 74399.1us
W20240426 13:20:09.537204 25030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.005s sys 0.003s
W20240426 13:20:09.537766 25030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.005s sys 0.003s
I20240426 13:20:09.468897 25026 rpcz_store.cc:276] 0426 13:20:07.251822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.841405 (+1589583us) service_pool.cc:225] Handling call
0426 13:20:08.969132 (+127727us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:08.969331 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:08.969633 (+ 302us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.147368 (+177735us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.147438 (+ 70us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:09.147510 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.147600 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:09.468385 (+320785us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54004169d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:09.468484 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70034,"prepare.queue_time_us":7240,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":38277,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":56766,"spinlock_wait_cycles":238727552,"thread_start_us":167560,"threads_started":1,"wal-append.queue_time_us":200751}
I20240426 13:20:09.451354 28025 raft_consensus.cc:2726] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240426 13:20:09.540343 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 66682.3us
I20240426 13:20:09.592891 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.612778 28035 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.007s sys 0.015s
I20240426 13:20:09.468711 27916 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:09.470371 27725 raft_consensus.cc:686] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.632519 24921 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:09.614804 28035 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s user 0.008s sys 0.015s
I20240426 13:20:09.539597 27677 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240426 13:20:09.617981 28146 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20240426 13:20:09.541384 28034 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.595006 28131 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.619814 28119 raft_consensus.cc:2781] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:09.642937 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.623142 27868 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240426 13:20:09.647603 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 288877us
I20240426 13:20:09.628693 27271 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:09.656554 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 164236us
W20240426 13:20:09.658011 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 208270us
I20240426 13:20:09.661624 27653 raft_consensus.cc:686] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.627964 27498 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:09.636770 27647 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:09.664754 25069 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.677179 25069 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.696481 25588 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.663535 28119 raft_consensus.cc:686] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.638126 27410 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.639467 27916 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.707268 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:09.712117 25028 raft_consensus.cc:1261] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:09.649179 28034 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.695387 27498 raft_consensus.cc:686] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:09.701143 27928 raft_consensus.cc:2741] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240426 13:20:09.731976 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.649853 28131 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:09.705399 27946 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:09.738065 25026 raft_consensus.cc:3037] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
W20240426 13:20:09.740531 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 98553.5us
I20240426 13:20:09.740756 25588 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.741344 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.742926 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.663439 27633 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.724370 28157 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:09.744586 27271 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240426 13:20:09.758067 28149 log.cc:924] Time spent T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20240426 13:20:09.760752 28148 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:09.761437 28148 raft_consensus.cc:2726] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.763062 28148 consensus_queue.cc:1046] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:09.764302 27498 consensus_queue.cc:237] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.743698 27653 consensus_queue.cc:237] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.767674 25026 raft_consensus.cc:2454] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:09.715284 27725 consensus_queue.cc:237] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.757301 24921 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.747408 27914 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:09.749955 27928 raft_consensus.cc:2741] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20240426 13:20:09.792467 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:09.732602 28155 consensus_queue.cc:1046] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.038s
I20240426 13:20:09.796197 24921 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:09.753504 27868 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:09.799122 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:09.792114 27914 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:09.807435 25026 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.808565 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.808451 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.809458 25104 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:09.810642 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.810777 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.811734 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.812196 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.812875 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.813222 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.813771 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:09.814524 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.814775 25103 raft_consensus.cc:2379] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.815301 25026 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:09.816021 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.815336 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.817490 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.820189 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:09.820461 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.821139 25104 raft_consensus.cc:2379] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:09.822155 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.824759 25028 raft_consensus.cc:1261] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240426 13:20:09.811749 25100 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:09.819808 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.829027 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.834111 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:09.832273 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:09.803526 27647 raft_consensus.cc:2726] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:09.838281 27647 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:09.838747 27647 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:09.840374 28147 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.204s user 0.000s sys 0.001s
I20240426 13:20:09.794847 27271 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:09.805476 25070 leader_election.cc:302] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:09.804984 27928 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:09.840550 28158 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
W20240426 13:20:09.867578 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 62187.3us
I20240426 13:20:09.752305 27738 consensus_queue.cc:1046] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
W20240426 13:20:09.985647 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 116235us
I20240426 13:20:09.840862 27914 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.046164 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.105235 25100 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:09.799679 28168 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:10.050099 27653 raft_consensus.cc:2781] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election won for term 7
I20240426 13:20:10.050228 27271 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.050592 28165 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.106958 27271 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.107105 28165 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.109125 27271 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.059712 28035 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.109925 28165 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:09.865339 28175 raft_consensus.cc:3032] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:10.112946 28175 raft_consensus.cc:729] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:09.756124 28157 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.114288 27916 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.115073 28157 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:10.097949 28172 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.301s user 0.000s sys 0.000s
W20240426 13:20:10.098847 28156 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.366s user 0.000s sys 0.000s
W20240426 13:20:10.059908 28144 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20240426 13:20:10.111261 28035 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.132884 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.112098 27647 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.060242 28154 consensus_queue.cc:1035] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.318s
I20240426 13:20:09.755218 28119 consensus_queue.cc:237] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:09.908692 27928 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:09.844530 27708 consensus_queue.cc:1035] T f04bd8996e6349879e25fca4e52dbda9 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.253s
W20240426 13:20:10.110602 28160 log.cc:924] Time spent T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.395s user 0.001s sys 0.000s
I20240426 13:20:10.218158 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:10.219028 24949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.062s sys 0.104s
W20240426 13:20:10.228965 24949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s user 0.062s sys 0.104s
W20240426 13:20:10.218569 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 541386us
I20240426 13:20:10.115093 28131 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.117244 28168 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.233803 25100 raft_consensus.cc:2396] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:10.230335 25069 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.239166 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 489508us
I20240426 13:20:10.242563 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:10.251989 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 435145us
I20240426 13:20:10.134344 27653 raft_consensus.cc:686] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:10.206566 28034 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.264951 25030 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2116) took 3029 ms. Trace:
I20240426 13:20:10.269601 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.217664 27410 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.229805 27928 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.274195 25030 rpcz_store.cc:276] 0426 13:20:07.235459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.614717 (+1379258us) service_pool.cc:225] Handling call
0426 13:20:08.615041 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:08.981907 (+366866us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:08.981953 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:08.982509 (+ 556us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.538237 (+555728us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.538297 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.538403 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.264648 (+726245us) raft_consensus.cc:1671] finished
0426 13:20:10.264678 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.264877 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":366345,"raft.queue_time_us":366665,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":89,"thread_start_us":555287,"threads_started":1,"wal-append.queue_time_us":663328}
W20240426 13:20:10.280336 24950 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.026s sys 0.088s
W20240426 13:20:10.257385 28158 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20240426 13:20:10.280058 28171 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:10.270448 28144 log.cc:924] Time spent T 499552c0877e46c5af41389ff1be3152 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20240426 13:20:10.283584 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.285411 24950 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.026s sys 0.088s
W20240426 13:20:10.286213 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 55977us
I20240426 13:20:10.296691 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.298241 27708 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.298674 27708 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.299105 27708 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.287701 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.300313 28171 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.302709 28171 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.299703 25069 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.303458 28171 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.304284 28178 raft_consensus.cc:2781] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:10.304905 28178 raft_consensus.cc:686] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:10.305969 28178 consensus_queue.cc:237] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:10.285812 28131 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.253680 28168 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:10.293270 27498 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.328645 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.329381 25622 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:10.331923 25100 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:10.333307 24954 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.000s sys 0.017s
W20240426 13:20:10.333637 24954 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s user 0.000s sys 0.017s
I20240426 13:20:10.334249 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2046) took 1097 ms. Trace:
I20240426 13:20:10.334578 24954 rpcz_store.cc:276] 0426 13:20:09.236583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.463369 (+226786us) service_pool.cc:225] Handling call
0426 13:20:09.474249 (+ 10880us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.474430 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:09.635235 (+160805us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.333961 (+698726us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:10.334008 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:10.334049 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.334109 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.334207 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":160401,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":3068,"thread_start_us":698269,"threads_started":1}
I20240426 13:20:10.290113 27868 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.308487 28131 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.342972 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:10.328768 28062 consensus_meta.cc:287] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.732s user 0.009s sys 0.004s
I20240426 13:20:10.337989 28171 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.347780 28173 raft_consensus.cc:3032] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:10.347776 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.346272 28159 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.401335 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.402052 24954 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
W20240426 13:20:10.402642 24951 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1705) took 1653 ms (client timeout 2023 ms). Trace:
I20240426 13:20:10.402980 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.398674 27725 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
W20240426 13:20:10.403079 24951 rpcz_store.cc:269] 0426 13:20:08.749117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.936254 (+187137us) service_pool.cc:225] Handling call
0426 13:20:10.402453 (+1466199us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b540004bd54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:10.402586 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":937559808}
I20240426 13:20:10.345440 27653 consensus_queue.cc:237] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.417557 28035 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.403900 27725 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.405037 24954 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:10.405081 27914 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.380510 28173 raft_consensus.cc:729] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.407692 24953 raft_consensus.cc:1226] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:10.411088 25617 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 616) took 1105 ms. Trace:
I20240426 13:20:10.426618 25617 rpcz_store.cc:276] 0426 13:20:09.305370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.305801 (+ 431us) service_pool.cc:225] Handling call
0426 13:20:09.306072 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.606993 (+300921us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.607035 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:09.607090 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:09.607206 (+ 116us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.410852 (+803646us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b5400473954. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.410897 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.411031 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21017,"raft.queue_time_us":98709,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6777,"spinlock_wait_cycles":105480960,"thread_start_us":193624,"threads_started":1}
I20240426 13:20:10.419488 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2045) took 1183 ms. Trace:
I20240426 13:20:10.429018 24949 rpcz_store.cc:276] 0426 13:20:09.236347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.456314 (+219967us) service_pool.cc:225] Handling call
0426 13:20:09.487120 (+ 30806us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.620367 (+133247us) raft_consensus.cc:1520] Early marking committed up to index 1
0426 13:20:09.620405 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.627898 (+ 7493us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.229388 (+601490us) raft_consensus.cc:1631] Marking committed up to 1
0426 13:20:10.229437 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.229548 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.419261 (+189713us) raft_consensus.cc:1671] finished
0426 13:20:10.419283 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.419417 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":132779,"raft.queue_time_us":142670,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":561563,"threads_started":1,"wal-append.queue_time_us":679133}
I20240426 13:20:10.431613 24949 raft_consensus.cc:1226] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:10.434195 28168 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.436841 28173 consensus_queue.cc:260] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.438385 28173 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:10.440642 28168 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.442044 27914 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.443941 28035 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.444509 24949 raft_consensus.cc:1226] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:10.447041 24951 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:10.448988 24949 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:10.456600 28131 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:10.455261 27725 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:10.464816 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 177928us
I20240426 13:20:10.467603 28096 raft_consensus.cc:3032] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:10.468154 28096 raft_consensus.cc:729] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:10.469142 28096 consensus_queue.cc:260] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.470655 28096 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:10.472999 28155 raft_consensus.cc:3032] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:10.473474 28155 raft_consensus.cc:729] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.474433 28155 consensus_queue.cc:260] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.481142 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:10.481920 24953 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.485502 25618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44520 (request call id 442) took 2483 ms. Trace:
I20240426 13:20:10.485991 25618 rpcz_store.cc:276] 0426 13:20:08.001735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.004654 (+ 2919us) service_pool.cc:225] Handling call
0426 13:20:08.117110 (+112456us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.356909 (+1239799us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.485213 (+1128304us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540056dbd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.485429 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1466532608}
I20240426 13:20:10.487936 25025 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51366 (request call id 2132) took 1487 ms. Trace:
I20240426 13:20:10.488420 25025 rpcz_store.cc:276] 0426 13:20:09.000707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.133056 (+132349us) service_pool.cc:225] Handling call
0426 13:20:10.487698 (+1354642us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400416ed4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:10.487857 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1029748608}
W20240426 13:20:10.493764 28170 log.cc:924] Time spent T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.382s user 0.001s sys 0.006s
I20240426 13:20:10.500429 28059 raft_consensus.cc:3032] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:10.500912 28059 raft_consensus.cc:729] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.501972 28059 consensus_queue.cc:260] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.508224 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 1.06032e+06us
I20240426 13:20:10.458868 28164 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.448077 28035 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:10.517866 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 51740.5us
I20240426 13:20:10.527087 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2051) took 1215 ms. Trace:
I20240426 13:20:10.526945 28175 consensus_queue.cc:260] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.527602 24952 rpcz_store.cc:276] 0426 13:20:09.312003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.469868 (+157865us) service_pool.cc:225] Handling call
0426 13:20:09.470158 (+ 290us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.590413 (+120255us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:09.590458 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.591101 (+ 643us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.662609 (+ 71508us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.662663 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:09.662756 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.493264 (+830508us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b2400274088. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d247ce4 00007fda8d245dfb 00007fda8d2d3e09 00007fda8d2d3a95 00007fda8d2d30dd 00007fda8ff822ee 00007fda8ff99a8c 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1
0426 13:20:10.521718 (+ 28454us) raft_consensus.cc:1671] finished
0426 13:20:10.521747 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.527012 (+ 5265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120884,"raft.queue_time_us":149771,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5985,"spinlock_wait_cycles":39615104,"thread_start_us":13900,"threads_started":1,"wal-append.queue_time_us":208108}
I20240426 13:20:10.517666 28059 raft_consensus.cc:3037] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.421885 28159 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.519037 28155 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:10.538573 28175 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:10.549113 28161 consensus_queue.cc:1046] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:10.549664 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 555) took 2570 ms. Trace:
I20240426 13:20:10.558085 25099 rpcz_store.cc:276] 0426 13:20:07.978835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:08.555421 (+576586us) service_pool.cc:225] Handling call
0426 13:20:08.555730 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:09.216481 (+660751us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.549417 (+1332936us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400168854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:10.549599 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1517808000}
I20240426 13:20:10.561517 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.560242 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:10.577529 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:10.578161 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 294628us
I20240426 13:20:10.578305 25101 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:10.581142 25104 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:10.587528 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.588989 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.589870 25030 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:10.591048 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.566079 27438 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.593418 28159 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:10.593855 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 79620.9us
I20240426 13:20:10.594758 28141 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.596486 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:10.566810 28104 consensus_meta.cc:287] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.939s user 0.008s sys 0.002s
I20240426 13:20:10.556373 28047 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.599344 25096 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:10.599453 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.623193 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.624042 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.642618 25618 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:10.626068 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.628484 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.629216 27438 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.579761 28173 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:10.631482 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.584556 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:10.555825 24953 raft_consensus.cc:2454] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
W20240426 13:20:10.694380 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 109914us
I20240426 13:20:10.695410 25588 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:20:10.621524 28036 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.566465 27928 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.630318 27810 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:10.727275 27928 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.726998 27810 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.728322 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 92589.5us
I20240426 13:20:10.730263 27810 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.730938 27810 consensus_queue.cc:1046] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:10.639405 27755 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.732254 24954 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:10.731973 27755 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.734450 27755 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.736378 27438 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.690313 25030 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:10.692916 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.741252 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.598220 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2031) took 1403 ms. Trace:
I20240426 13:20:10.743047 24950 rpcz_store.cc:276] 0426 13:20:09.194308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.428575 (+234267us) service_pool.cc:225] Handling call
0426 13:20:09.428860 (+ 285us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:09.443909 (+ 15049us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:09.463310 (+ 19401us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.332824 (+869514us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:10.342928 (+ 10104us) log.cc:842] Serialized 52 byte log entry
0426 13:20:10.343093 (+ 165us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:10.596722 (+253629us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.597257 (+ 535us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.597450 (+ 193us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.597528 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.597587 (+ 59us) raft_consensus.cc:1671] finished
0426 13:20:10.597605 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.597778 (+ 173us) inbound_call.cc:173] Queueing success response
0426 13:20:10.644503 (+ 46725us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b4800000b58. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda951bf2e1 00007fda951b63d7 00007fda951b626a 00007fda952aa43f 00007fda952aa107 00007fda952be60c 00007fda952be5b8 00007fda952be538 00007fda952be4f3 00007fda952bd79c 00007fda942408d6
Metrics: {"mutex_wait_us":15919,"prepare.queue_time_us":154358,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":107,"raft.queue_time_us":30604,"raft.run_cpu_time_us":12343,"raft.run_wall_time_us":1249635,"spinlock_wait_cycles":2915200,"thread_start_us":924256,"threads_started":2,"wal-append.queue_time_us":804428}
I20240426 13:20:10.625576 27900 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.698426 25104 raft_consensus.cc:2396] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240426 13:20:10.700135 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:10.640509 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.747660 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 107220us
I20240426 13:20:10.587635 28188 consensus_queue.cc:1046] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20240426 13:20:10.748800 24921 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.738925 28036 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.749984 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.751206 28188 raft_consensus.cc:3032] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:10.639855 28104 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.716158 28193 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:10.752152 27647 raft_consensus.cc:2726] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.752256 28193 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.752843 28193 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:10.752914 27647 raft_consensus.cc:2726] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.687834 28047 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.753329 28104 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.753628 27647 raft_consensus.cc:2726] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.754850 28047 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:10.754866 24921 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.686751 28190 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.724941 28184 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.755720 28190 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.756258 28035 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.756424 28184 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.757795 28190 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.718482 28173 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.751628 28188 raft_consensus.cc:729] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.758523 28184 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.724723 27708 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.759181 28173 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.759385 28188 consensus_queue.cc:260] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.752864 28185 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240426 13:20:10.760556 28188 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.760600 27708 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.597060 27847 consensus_queue.cc:1046] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20240426 13:20:10.753080 28176 raft_consensus.cc:2781] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:10.557178 28164 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.761790 28176 raft_consensus.cc:686] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.762053 27847 raft_consensus.cc:3032] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:10.762403 27847 raft_consensus.cc:729] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:10.762662 28176 consensus_queue.cc:237] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:10.718294 28179 log.cc:924] Time spent T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20240426 13:20:10.763185 27847 consensus_queue.cc:260] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.763674 28164 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.643945 27725 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.765497 24921 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.765633 24953 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:10.766403 25099 raft_consensus.cc:1226] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:10.766975 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.767200 24953 raft_consensus.cc:1226] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 2. Current term is 3. Ops: []
I20240426 13:20:10.726085 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.769387 24921 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.771320 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 65447.3us
I20240426 13:20:10.772997 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:10.774755 25590 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.011s sys 0.024s
W20240426 13:20:10.776335 25590 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.012s sys 0.024s
I20240426 13:20:10.775771 24921 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.785705 25617 raft_consensus.cc:1226] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:10.779031 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.789963 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.791019 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.778930 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.717219 28062 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.794229 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.798875 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.630745 28025 consensus_queue.cc:1035] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.135s
I20240426 13:20:10.644587 28159 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:10.808437 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.808665 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.780376 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.799988 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.774417 28168 raft_consensus.cc:2726] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.808166 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.816185 24950 raft_consensus.cc:1226] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:10.810827 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:10.778594 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:10.813194 25101 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:10.816748 25096 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:10.774609 28124 raft_consensus.cc:2726] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:10.764364 27725 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.790287 27647 raft_consensus.cc:2726] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240426 13:20:10.797148 28076 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240426 13:20:10.644956 27868 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.821910 24920 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.833501 24920 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.777896 27928 raft_consensus.cc:2726] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:10.827425 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:10.835611 25031 raft_consensus.cc:2379] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:10.834370 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.837885 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.839381 24921 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.640916 28141 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.819855 27847 raft_consensus.cc:3037] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:10.837057 28034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.011s sys 0.028s
I20240426 13:20:10.862988 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.848214 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 1.24937e+06us
W20240426 13:20:10.856511 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 83615.5us
I20240426 13:20:10.826782 27900 raft_consensus.cc:2726] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:10.779220 28175 consensus_queue.cc:1046] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20240426 13:20:10.860728 25030 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:10.892398 25022 raft_consensus.cc:1261] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240426 13:20:10.861150 24950 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:10.863062 25589 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.866786 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:10.894237 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:10.896281 27708 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.896441 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2029) took 1702 ms. Trace:
I20240426 13:20:10.896703 27708 raft_consensus.cc:482] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.897147 27708 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.900589 28178 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:10.901120 28178 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:10.901515 28178 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:10.896868 24955 rpcz_store.cc:276] 0426 13:20:09.193774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.407113 (+213339us) service_pool.cc:225] Handling call
0426 13:20:09.424843 (+ 17730us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:09.425262 (+ 419us) raft_consensus.cc:1520] Early marking committed up to index 2
0426 13:20:09.425596 (+ 334us) log.cc:842] Serialized 14 byte log entry
0426 13:20:09.633794 (+208198us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:09.643538 (+ 9744us) log.cc:842] Serialized 26 byte log entry
0426 13:20:09.643696 (+ 158us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:09.643906 (+ 210us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.477159 (+833253us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ece78 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:10.517907 (+ 40748us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ecfa4 00007fda8d3f0351 00007fda8d3f0309 00007fda8d3f0299 00007fda8d3f0261 00007fda8d3ef55d 00007fda96660d34 00007fda96660b69 00007fda8d3cf5f6
0426 13:20:10.556430 (+ 38523us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18000e2ea0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:10.896130 (+339700us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.896208 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:10.896251 (+ 43us) raft_consensus.cc:1671] finished
0426 13:20:10.896266 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:10.896387 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20883,"prepare.queue_time_us":4815,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"raft.queue_time_us":73163,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":32969,"spinlock_wait_cycles":37390720,"thread_start_us":526033,"threads_started":2,"wal-append.queue_time_us":216036}
I20240426 13:20:10.905268 28104 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:10.908466 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:10.910487 27498 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:10.910976 27498 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.911422 27498 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:10.868580 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.912720 27438 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240426 13:20:10.912878 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 44410.6us
I20240426 13:20:10.913151 27438 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:10.913538 27438 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:10.913674 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:10.914865 24995 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:10.916476 28025 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:10.916997 28025 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.923393 28025 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:10.862691 28034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.011s sys 0.028s
I20240426 13:20:10.847841 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:10.928678 24995 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:10.808403 28062 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.851152 27647 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:10.846411 28182 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.854106 28036 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:10.861729 28124 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.834936 28059 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20240426 13:20:10.938593 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 75614.1us
I20240426 13:20:10.941829 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:10.869943 27653 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:10.957892 24995 leader_election.cc:302] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:10.944581 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 96833us
I20240426 13:20:10.928591 28097 raft_consensus.cc:2726] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240426 13:20:10.822697 28159 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:10.945876 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:10.955595 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:10.929255 28061 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.003s sys 0.027s
I20240426 13:20:10.967728 27438 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.937494 28139 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.846546 27868 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.940513 28152 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:10.970997 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:10.937956 27900 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:10.963660 28161 raft_consensus.cc:2726] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:10.952200 27810 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:10.957258 25616 raft_consensus.cc:1261] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Refusing update from remote peer 1e87e7bb414642aeb72fe0d2ff72cc2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:11.001439 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.976889 28061 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.003s sys 0.028s
I20240426 13:20:10.977612 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:10.985383 28096 consensus_meta.cc:287] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.001s
I20240426 13:20:10.999399 27438 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.002008 25100 raft_consensus.cc:2396] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:10.956630 28035 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.022629 28035 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:10.981415 27647 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.024335 27647 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.026697 27647 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.027777 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.029407 28035 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.004179 25023 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:11.007131 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:10.981562 28175 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:10.960659 28036 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20240426 13:20:11.057749 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.000075 27810 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240426 13:20:11.068707 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 159164us
I20240426 13:20:11.015301 28195 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.001087 28206 raft_consensus.cc:2726] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:11.083345 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:11.011518 27755 consensus_queue.cc:1035] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 13:20:11.001367 28173 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:11.113698 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.114574 25102 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
W20240426 13:20:11.087356 28204 log.cc:924] Time spent T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20240426 13:20:11.088424 28199 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20240426 13:20:11.092080 27498 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.100085 28159 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.100709 27725 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.069211 27708 raft_consensus.cc:504] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.083969 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:10.996002 27900 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.071713 27810 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:11.122246 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 156269us
I20240426 13:20:11.083052 28061 raft_consensus.cc:3032] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.156085 25030 raft_consensus.cc:3032] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.156762 25030 raft_consensus.cc:729] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.170068 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.063933 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.167110 25030 consensus_queue.cc:260] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
W20240426 13:20:11.170118 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 136151us
I20240426 13:20:11.173345 25030 raft_consensus.cc:3037] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.149267 28061 raft_consensus.cc:729] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.085222 28195 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.212491 25069 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.157155 28141 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.216384 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.165891 28062 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.213583 28139 consensus_queue.cc:1035] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
W20240426 13:20:11.226650 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 86663.4us
I20240426 13:20:11.209677 28059 raft_consensus.cc:2781] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.244863 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.287846 28197 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:11.287849 28059 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.288585 28197 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.288923 28059 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.289194 28197 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:11.306846 24995 leader_election.cc:302] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.309203 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.306846 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.310655 25031 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.313987 25025 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.216106 27868 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
W20240426 13:20:11.317052 28193 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.564s user 0.012s sys 0.000s
I20240426 13:20:11.317129 27438 raft_consensus.cc:2726] T 256415e42b6c444387115ab766a5a16c P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:11.317752 28193 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.319536 25620 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 617) took 2000 ms. Trace:
I20240426 13:20:11.320225 27708 leader_election.cc:288] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.321761 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 564) took 1077 ms. Trace:
I20240426 13:20:11.319977 25620 rpcz_store.cc:276] 0426 13:20:09.319137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.319273 (+ 136us) service_pool.cc:225] Handling call
0426 13:20:10.478998 (+1159725us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:10.479239 (+ 241us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:10.546471 (+ 67232us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:10.546518 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:10.547031 (+ 513us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.994131 (+447100us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:10.999027 (+ 4896us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.999501 (+ 474us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.999631 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:11.319248 (+319617us) raft_consensus.cc:1671] finished
0426 13:20:11.319353 (+ 105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:11.319482 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44740,"prepare.queue_time_us":4275,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":49566,"raft.run_cpu_time_us":7220,"raft.run_wall_time_us":9764,"spinlock_wait_cycles":16928000,"thread_start_us":446681,"threads_started":1,"wal-append.queue_time_us":769862}
I20240426 13:20:11.322168 24948 rpcz_store.cc:276] 0426 13:20:10.244867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.245081 (+ 214us) service_pool.cc:225] Handling call
0426 13:20:10.245485 (+ 404us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:10.268552 (+ 23067us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:10.474195 (+205643us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.829917 (+355722us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.831515 (+ 1598us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:10.832025 (+ 510us) log.cc:842] Serialized 26 byte log entry
0426 13:20:10.832191 (+ 166us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:10.854256 (+ 22065us) log.cc:842] Serialized 14 byte log entry
0426 13:20:10.859748 (+ 5492us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:10.859845 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:11.321590 (+461745us) raft_consensus.cc:1671] finished
0426 13:20:11.321609 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:11.321708 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31177,"prepare.queue_time_us":221048,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":70,"raft.queue_time_us":55364,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":44584,"spinlock_wait_cycles":37149824,"thread_start_us":549016,"threads_started":2,"wal-append.queue_time_us":468032}
W20240426 13:20:11.327384 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 111712us
I20240426 13:20:11.340243 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:11.341156 25619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 618) took 2006 ms (client timeout 1977 ms). Trace:
I20240426 13:20:11.341742 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 569) took 1086 ms. Trace:
W20240426 13:20:11.341545 25619 rpcz_store.cc:269] 0426 13:20:09.341009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.348383 (+ 7374us) service_pool.cc:225] Handling call
0426 13:20:11.340079 (+1991696us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:11.341087 (+ 1008us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13832960}
I20240426 13:20:11.342250 24954 rpcz_store.cc:276] 0426 13:20:10.256909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.435245 (+178336us) service_pool.cc:225] Handling call
0426 13:20:10.435572 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.732691 (+297119us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.341534 (+608843us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54002cb5d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:11.341690 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":681811200}
I20240426 13:20:11.355154 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.356839 25103 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:11.358534 28195 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.359616 25097 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:11.362767 25025 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:11.365612 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:11.356211 27923 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:11.366887 28215 log.cc:924] Time spent T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
W20240426 13:20:11.367220 28207 log.cc:924] Time spent T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20240426 13:20:11.369597 28139 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:11.372553 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.351423 28193 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.375500 27853 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.376328 28237 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.377274 25029 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 406) took 3919 ms. Trace:
I20240426 13:20:11.377671 25029 rpcz_store.cc:276] 0426 13:20:07.458336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.239402 (+1781066us) service_pool.cc:225] Handling call
0426 13:20:09.468297 (+228895us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:09.489039 (+ 20742us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.377091 (+1888052us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54003f7d50. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.377220 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":525655936}
I20240426 13:20:11.379552 24996 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:11.381767 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.384543 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.385831 25030 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240426 13:20:11.380123 27498 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.401710 25097 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.322207 28178 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.411464 28178 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.399729 28237 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.414212 28237 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.415270 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.417886 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:11.403187 27847 consensus_meta.cc:287] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.000s
I20240426 13:20:11.429503 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.430570 25025 raft_consensus.cc:1261] T a6a69cdcf2ee4768a20888dd974c6ba7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20240426 13:20:11.430770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121666us
I20240426 13:20:11.389889 28139 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.432729 24995 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:11.463460 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.463912 25622 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:11.465741 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.468235 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.414201 27900 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.427286 28227 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.475024 28197 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.477246 25028 raft_consensus.cc:3032] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:11.477623 25028 raft_consensus.cc:729] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.478099 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.478694 25026 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.478359 25028 consensus_queue.cc:260] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.479463 25028 raft_consensus.cc:3037] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.476879 28059 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.483186 28059 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.486879 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.488718 28197 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:11.489585 28163 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 1.428s user 0.006s sys 0.000s
I20240426 13:20:11.496884 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.497596 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
W20240426 13:20:11.497897 28216 log.cc:924] Time spent T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
I20240426 13:20:11.498582 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.499230 24949 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
W20240426 13:20:11.500591 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 418494us
I20240426 13:20:11.433394 28139 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.503289 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.504258 24952 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.506477 24949 raft_consensus.cc:1226] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.507923 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.508567 24953 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:11.509064 25102 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:11.512319 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.512909 24952 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.513809 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.514422 24949 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.546463 24949 raft_consensus.cc:1261] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240426 13:20:11.518306 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.550963 24953 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
W20240426 13:20:11.502120 28131 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.015s sys 0.041s
I20240426 13:20:11.554283 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.554905 24949 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.555670 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.426983 28193 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
W20240426 13:20:11.546864 28188 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.786s user 0.008s sys 0.004s
I20240426 13:20:11.558194 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.557741 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
W20240426 13:20:11.557315 28131 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.006s user 0.016s sys 0.041s
I20240426 13:20:11.433576 28212 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
I20240426 13:20:11.508352 27928 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.503588 28237 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.508857 27799 raft_consensus.cc:3032] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:11.609545 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.610373 24955 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:11.563267 25028 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.565779 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:11.520514 25026 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.502900 27900 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.567114 25588 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.616403 24953 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:11.616593 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 49587.7us
I20240426 13:20:11.549696 27932 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.617368 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.618019 24953 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:11.618835 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.619514 24953 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.517165 28168 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.548920 28197 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.620839 28168 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.621662 28197 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.508985 28139 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.604709 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.624289 28176 consensus_queue.cc:1046] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:11.625252 27853 raft_consensus.cc:2726] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240426 13:20:11.562602 28194 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20240426 13:20:11.553557 27810 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.630033 27810 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.630770 25024 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:11.545428 28059 consensus_queue.cc:1046] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240426 13:20:11.614900 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.623267 24949 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:11.613515 25028 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51366 (request call id 2156) took 1034 ms. Trace:
I20240426 13:20:11.632107 28059 raft_consensus.cc:3032] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:11.632614 28059 raft_consensus.cc:729] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:11.632369 25028 rpcz_store.cc:276] 0426 13:20:10.579506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.579678 (+ 172us) service_pool.cc:225] Handling call
0426 13:20:10.580180 (+ 502us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:11.477150 (+896970us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b54002894a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda902a78b3 00007fda90319490 00007fda9031a501 00007fda9031706f 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c
0426 13:20:11.613211 (+136061us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.613449 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158845696}
I20240426 13:20:11.633855 28059 consensus_queue.cc:260] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:11.635774 27943 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20240426 13:20:11.613380 27928 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.568955 28126 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.609027 27799 raft_consensus.cc:729] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:11.645866 24948 raft_consensus.cc:1261] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:11.648808 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.612728 28025 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.469471 28239 consensus_queue.cc:1046] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.181s
I20240426 13:20:11.658607 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:11.659174 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 226525us
I20240426 13:20:11.660122 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.653040 24953 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:11.661108 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.661810 24955 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.644655 28193 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.667901 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.667541 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.668073 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.662444 24995 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.671483 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.672279 25069 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:20:11.680684 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 211017us
I20240426 13:20:11.681862 25069 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.683300 25069 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.684705 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:11.686058 25069 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.686133 25589 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.692770 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.693378 24953 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.666460 27810 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.614512 28237 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.708549 24955 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:11.709746 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.711477 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.503259 28227 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.662011 27928 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.721648 25097 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:11.724812 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.725152 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.651439 27900 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.694576 27869 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:11.691890 28059 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.725536 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:11.733662 24948 raft_consensus.cc:1261] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:11.734556 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.735121 24948 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:11.697011 28212 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:11.736655 24995 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:11.696322 28173 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:11.729189 25069 leader_election.cc:302] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.696316 28034 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:11.730312 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.697010 28178 raft_consensus.cc:2781] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:11.663372 28025 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.696388 27868 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:11.710729 27916 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.744993 27916 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.745388 27916 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.746652 27900 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.739898 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:11.748559 25097 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.749588 28222 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.749982 24948 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:11.696725 27498 raft_consensus.cc:2781] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.740418 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.725908 28168 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.752185 25069 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.772714 24953 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:11.779435 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.696875 28175 consensus_queue.cc:1035] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20240426 13:20:11.682869 28193 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.789943 24953 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:11.744457 27451 raft_consensus.cc:2781] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.731400 28224 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.700536 27810 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.700656 28161 consensus_queue.cc:1035] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20240426 13:20:11.732340 27869 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.652415 28126 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.740449 28034 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.739898 28178 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.750180 28222 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:11.752384 27946 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:11.798617 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46518.8us
I20240426 13:20:11.751066 27498 raft_consensus.cc:482] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.754765 28237 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.720167 28166 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.070s
I20240426 13:20:11.810566 28222 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.816798 28025 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.800925 27451 raft_consensus.cc:686] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.817955 27451 consensus_queue.cc:237] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.800015 27810 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.821293 28243 raft_consensus.cc:2781] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:11.821739 28243 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.822096 28243 raft_consensus.cc:3037] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.804011 28034 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.824484 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.825377 25026 raft_consensus.cc:2454] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:11.711624 27799 consensus_queue.cc:260] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.826964 27799 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.807986 28178 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:11.803674 28184 raft_consensus.cc:2781] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.832692 28184 raft_consensus.cc:686] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:11.833604 28184 consensus_queue.cc:237] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.834626 25622 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44512 (request call id 638) took 1056 ms. Trace:
I20240426 13:20:11.835063 25622 rpcz_store.cc:276] 0426 13:20:10.778242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.778364 (+ 122us) service_pool.cc:225] Handling call
0426 13:20:11.834471 (+1056107us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54006207d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.834574 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1485646208}
I20240426 13:20:11.836897 28227 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.838750 28126 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:11.804255 27869 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:11.841033 28193 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.841328 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.842504 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.843051 25102 raft_consensus.cc:2454] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:11.812393 28197 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:11.814325 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.739300 24955 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:11.767555 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58866 (request call id 1537) took 1144 ms. Trace:
I20240426 13:20:11.870800 25100 rpcz_store.cc:276] 0426 13:20:10.622648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.622841 (+ 193us) service_pool.cc:225] Handling call
0426 13:20:11.767322 (+1144481us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b540001bdd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:11.767481 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":722548480}
I20240426 13:20:11.872107 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.750802 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.873462 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.750944 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.874130 24955 raft_consensus.cc:2354] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:11.874239 24948 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 10: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:11.752660 28233 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20240426 13:20:11.875799 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.876863 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:11.877960 25028 raft_consensus.cc:2396] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:11.868399 25069 leader_election.cc:302] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.868649 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:11.843725 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:11.763958 28202 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.768s user 0.000s sys 0.002s
I20240426 13:20:11.807346 25097 raft_consensus.cc:2454] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:11.813324 27946 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.882719 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:11.815515 27498 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.880532 25102 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:20:11.903520 24948 raft_consensus.cc:1261] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:11.904222 25100 raft_consensus.cc:2396] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240426 13:20:11.888352 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.767738 27868 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.765384 28240 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.916734 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:11.884858 28126 raft_consensus.cc:2781] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:11.926095 25104 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:11.923487 26679 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:11.887287 27725 raft_consensus.cc:2781] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:11.905498 27932 raft_consensus.cc:2726] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:11.927572 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:11.761216 27916 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.800573 28224 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:11.914932 27911 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:11.814669 28247 consensus_queue.cc:1046] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.114s
I20240426 13:20:11.917048 28240 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.952889 27911 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.953498 27911 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:11.954223 28025 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:11.954767 28025 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.921268 28222 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:11.957257 28246 log.cc:924] Time spent T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240426 13:20:11.912220 27868 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.923059 27738 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.906205 28035 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20240426 13:20:11.965173 25102 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:11.953097 28247 raft_consensus.cc:3032] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:11.965685 28247 raft_consensus.cc:729] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:11.966607 28247 consensus_queue.cc:260] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.967191 28034 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.887754 28055 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:11.906214 27946 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:11.971360 27869 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:11.925101 28097 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:11.972481 28097 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.972856 28097 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:11.978354 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 575) took 1416 ms. Trace:
I20240426 13:20:11.978699 28224 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:11.978749 25099 rpcz_store.cc:276] 0426 13:20:10.562088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.562703 (+ 615us) service_pool.cc:225] Handling call
0426 13:20:10.562965 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.766867 (+203902us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:11.978131 (+1211264us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54001b3d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:11.978286 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467356928}
I20240426 13:20:11.924918 28126 raft_consensus.cc:482] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:11.982947 28126 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:11.964628 27738 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:11.988487 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 249667us
I20240426 13:20:11.990602 25615 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3.
I20240426 13:20:11.951021 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:11.999034 25615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44496 (request call id 619) took 2650 ms (client timeout 1891 ms). Trace:
W20240426 13:20:11.999539 25615 rpcz_store.cc:269] 0426 13:20:09.348958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.349111 (+ 153us) service_pool.cc:225] Handling call
0426 13:20:11.565624 (+2216513us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b58000e1698. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda95259e20 00007fda95259704 00007fda8e93dbd4 00007fda8e93db62
0426 13:20:11.998923 (+433299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12372352}
I20240426 13:20:11.966500 28175 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:11.926621 28212 raft_consensus.cc:686] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.020077 24956 raft_consensus.cc:1226] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:12.022804 24954 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 5: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:12.024708 24947 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2048) took 2770 ms (client timeout 1977 ms). Trace:
W20240426 13:20:12.025259 24947 rpcz_store.cc:269] 0426 13:20:09.247119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.263480 (+ 16361us) service_pool.cc:225] Handling call
0426 13:20:12.017768 (+2754288us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b5400366ad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:12.017917 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78129024}
I20240426 13:20:12.027160 24947 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:12.028924 24947 raft_consensus.cc:1226] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: []
I20240426 13:20:11.874841 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:12.030761 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 294180us
I20240426 13:20:12.024708 24954 raft_consensus.cc:1261] T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240426 13:20:12.060598 25105 raft_consensus.cc:1226] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:12.192023 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:12.193768 24996 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:12.194790 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 284688us
I20240426 13:20:11.995870 27725 raft_consensus.cc:686] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:12.209962 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.209745 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.216872 25028 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:12.218391 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.218591 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:12.219031 25028 raft_consensus.cc:2454] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:12.018088 28096 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.200873 24955 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:12.200234 25619 raft_consensus.cc:1226] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Rejecting Update request from peer 1e87e7bb414642aeb72fe0d2ff72cc2e for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:12.209471 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:11.958647 28240 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:12.197521 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 134345us
I20240426 13:20:12.212383 27869 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.190057 28247 raft_consensus.cc:3037] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:12.196766 27271 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.221025 28238 consensus_queue.cc:1046] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:12.222187 28233 consensus_queue.cc:1046] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:12.224788 28047 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
W20240426 13:20:12.226691 28241 log.cc:924] Time spent T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.749s user 0.003s sys 0.000s
I20240426 13:20:12.229775 27932 raft_consensus.cc:2781] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election won for term 9
W20240426 13:20:12.231081 28139 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.608s user 0.013s sys 0.002s
I20240426 13:20:12.232460 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1162) took 1292 ms. Trace:
I20240426 13:20:12.288682 27799 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:12.244985 28249 log.cc:924] Time spent T a6a69cdcf2ee4768a20888dd974c6ba7 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20240426 13:20:12.246099 27738 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.248994 27410 raft_consensus.cc:3032] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Stepping down as leader of term 2
I20240426 13:20:12.250559 27911 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.252156 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.196640 28168 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.254348 28178 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.210516 28212 consensus_queue.cc:237] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.213644 27708 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:12.196702 28049 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.256316 27946 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.213292 28188 raft_consensus.cc:482] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.060817 28171 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.271641 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.197908 24995 leader_election.cc:302] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:12.346909 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 149459us
I20240426 13:20:12.347929 24995 leader_election.cc:398] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:12.348269 24995 leader_election.cc:401] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:12.349103 25097 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:12.287418 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.350596 25024 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.023947 28175 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.292881 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:12.259543 28222 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.220160 27923 consensus_queue.cc:1046] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20240426 13:20:12.163779 28243 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.236457 28142 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.306525 25101 rpcz_store.cc:276] 0426 13:20:10.939800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.939956 (+ 156us) service_pool.cc:225] Handling call
0426 13:20:12.231757 (+1291801us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b54003edfd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029cc70 00007fda903200ba 00007fda9525d8de 00007fda8e9507d7 00007fda8e950762 00007fda8e950691 00007fda8e95060c 00007fda8e94f8c2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:12.232379 (+ 622us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1083905280}
I20240426 13:20:12.329960 24949 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:12.262223 28096 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.303315 28025 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.311913 27410 raft_consensus.cc:729] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:12.345909 24955 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 5.
I20240426 13:20:12.331070 27932 raft_consensus.cc:686] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.300004 27271 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.354723 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.380115 25099 raft_consensus.cc:1226] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:12.312587 27869 raft_consensus.cc:3032] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Stepping down as leader of term 4
I20240426 13:20:12.365727 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.381711 24955 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:12.357955 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 65162.3us
I20240426 13:20:12.382961 25589 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:12.354516 25103 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:12.379998 24950 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.384495 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:12.384771 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.306478 28047 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.306054 27810 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.359871 28055 raft_consensus.cc:3032] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:12.359264 27923 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:12.308521 28139 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.396235 24949 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:12.402094 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.357507 28254 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.421978 24955 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:12.452298 24853 catalog_manager.cc:5484] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 7, leader changed from 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4) to 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1). New cstate: current_term: 7 leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
W20240426 13:20:12.423282 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 383898us
I20240426 13:20:12.406956 24947 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:12.352917 28258 raft_consensus.cc:2781] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader election won for term 5
I20240426 13:20:12.428412 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.459051 24949 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:12.428530 25024 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 9.
I20240426 13:20:12.311831 27799 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.365725 28096 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.365109 27410 consensus_queue.cc:260] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.364449 28025 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:12.460829 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:12.461136 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 566) took 2213 ms. Trace:
I20240426 13:20:12.387269 28148 raft_consensus.cc:2726] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:12.433918 27911 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.395749 28154 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:12.434792 28178 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.435446 28124 consensus_queue.cc:1046] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240426 13:20:12.359707 28142 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.437166 27350 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:12.439122 27946 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.360517 28243 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.356158 28175 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.353938 27868 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.427467 27923 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.444718 28139 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.348934 28049 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.351406 28188 raft_consensus.cc:504] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.447649 27271 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.383793 27869 raft_consensus.cc:729] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:12.338014 28168 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.376363 28257 raft_consensus.cc:2726] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6.
I20240426 13:20:12.351161 28171 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.422827 28055 raft_consensus.cc:729] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:12.456144 28097 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:12.600203 28256 log.cc:924] Time spent T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.252s user 0.000s sys 0.001s
W20240426 13:20:12.635542 28264 log.cc:924] Time spent T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.292s user 0.001s sys 0.000s
I20240426 13:20:12.468751 28155 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:12.468508 28154 raft_consensus.cc:482] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.514452 28258 raft_consensus.cc:686] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.813068 28269 raft_consensus.cc:3032] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:12.849442 28154 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:12.869872 28269 raft_consensus.cc:729] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:12.809427 28155 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:12.871098 28155 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:12.873421 28168 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:12.456445 28254 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.871223 28269 consensus_queue.cc:260] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:12.907199 28262 log.cc:924] Time spent T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20240426 13:20:12.911033 28269 raft_consensus.cc:3037] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:12.457369 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 186295us
W20240426 13:20:12.457120 24921 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.23.61.2:43671 timed out after 1.780s (SENT) [suppressed 22 similar messages]
W20240426 13:20:12.950306 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 492993us
I20240426 13:20:12.960603 24950 raft_consensus.cc:2454] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:12.461504 24951 rpcz_store.cc:276] 0426 13:20:10.247269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.419217 (+171948us) service_pool.cc:225] Handling call
0426 13:20:10.423314 (+ 4097us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.447416 (+ 24102us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:12.460930 (+2013514us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54001d06d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:12.461086 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54131072}
I20240426 13:20:12.455122 24955 raft_consensus.cc:1226] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20240426 13:20:12.968581 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1715) took 1285 ms. Trace:
I20240426 13:20:12.968449 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:12.969300 24955 rpcz_store.cc:276] 0426 13:20:11.682570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.454525 (+771955us) service_pool.cc:225] Handling call
0426 13:20:12.454887 (+ 362us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:12.967696 (+512809us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:12.968348 (+ 652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:12.971004 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.971248 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:12.972213 24951 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:12.985086 24995 leader_election.cc:302] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:12.445400 25023 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:12.521301 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.472131 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:12.995191 25022 raft_consensus.cc:3032] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:12.995919 25027 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:12.996491 25025 raft_consensus.cc:3032] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:12.997025 25025 raft_consensus.cc:729] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:12.997668 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.998101 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.998602 25025 consensus_queue.cc:260] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.515002 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.000005 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:12.515523 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 302047us
I20240426 13:20:13.001441 25031 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:12.996862 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:12.995834 25022 raft_consensus.cc:729] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.004802 25022 consensus_queue.cc:260] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.995095 25024 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.006085 25022 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.008011 25030 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:13.009765 25025 raft_consensus.cc:3037] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:12.309597 27725 consensus_queue.cc:237] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:12.467088 28096 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.067140 28161 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.067344 28139 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:12.773460 28047 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.454784 28126 raft_consensus.cc:504] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.088325 28055 consensus_queue.cc:260] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.090685 28055 raft_consensus.cc:3037] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:12.788970 27923 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.092473 28175 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.076751 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1849) took 1388 ms. Trace:
I20240426 13:20:13.093475 28175 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.093864 25105 rpcz_store.cc:276] 0426 13:20:11.687896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.688085 (+ 189us) service_pool.cc:225] Handling call
0426 13:20:11.711277 (+ 23192us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.193498 (+482221us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.075938 (+882440us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540039e254. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.076435 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":787088640}
I20240426 13:20:13.096117 28168 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.096943 28168 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.101373 24948 raft_consensus.cc:2379] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:13.100734 28258 consensus_queue.cc:237] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.101536 27869 consensus_queue.cc:260] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.103256 27869 raft_consensus.cc:3037] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:13.103627 27868 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.088752 25100 raft_consensus.cc:1226] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:13.104441 28209 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.105152 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.105556 28209 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.107583 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.108310 27799 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:12.525749 28025 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.708926 27708 raft_consensus.cc:482] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:12.467224 27932 consensus_queue.cc:237] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:12.722608 27810 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:12.465270 28059 consensus_queue.cc:1159] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "46106ff992984d33bb4a4d8f30162639" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20240426 13:20:13.076895 28097 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:12.836330 28124 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.200695 28097 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.202397 28097 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.133232 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.108536 25030 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:13.206739 28139 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.134836 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 579) took 2281 ms. Trace:
I20240426 13:20:13.139349 25102 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1852) took 1412 ms. Trace:
W20240426 13:20:13.112262 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 127757us
I20240426 13:20:13.114872 28257 raft_consensus.cc:2781] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:13.131417 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.107599 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
W20240426 13:20:13.119269 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49506.7us
I20240426 13:20:13.116880 24950 raft_consensus.cc:1226] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20240426 13:20:13.143826 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 141971us
I20240426 13:20:13.113945 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41288 (request call id 1734) took 1378 ms. Trace:
I20240426 13:20:13.134212 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.099871 28168 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.138139 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.288408 24953 rpcz_store.cc:276] 0426 13:20:11.735466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.779118 (+ 43652us) service_pool.cc:225] Handling call
0426 13:20:13.113404 (+1334286us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b540038e0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.113700 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22328192}
I20240426 13:20:13.132427 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.304203 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1719) took 1595 ms. Trace:
I20240426 13:20:13.114598 28248 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:13.313668 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.313956 28248 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.314517 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.315353 28139 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.109059 28206 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.316485 28206 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:13.185107 27451 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.308666 24950 rpcz_store.cc:276] 0426 13:20:11.701597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.115815 (+1414218us) service_pool.cc:225] Handling call
0426 13:20:13.116482 (+ 667us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.291100 (+174618us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.297258 (+ 6158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.291517 25096 rpcz_store.cc:276] 0426 13:20:10.853530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.853976 (+ 446us) service_pool.cc:225] Handling call
0426 13:20:10.865672 (+ 11696us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.993857 (+128185us) raft_consensus.cc:1520] Early marking committed up to index 7
0426 13:20:11.187286 (+193429us) log.cc:842] Serialized 14 byte log entry
0426 13:20:11.961399 (+774113us) spinlock_profiling.cc:243] Waited 5.24 ms on lock 0x7b18003996a0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3ec652 00007fda8ff9843b 00007fda8ff9dee3 00007fda8ff9e3fb 00007fda90321a1a
0426 13:20:12.198612 (+237213us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:12.198684 (+ 72us) raft_consensus.cc:1631] Marking committed up to 7
0426 13:20:12.198745 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.134197 (+935452us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540012cad4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.134280 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.134601 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":205454,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":1962,"raft.queue_time_us":12932,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":45598,"spinlock_wait_cycles":306289792,"thread_start_us":393690,"threads_started":2,"wal-append.queue_time_us":1093287}
I20240426 13:20:13.109306 28237 raft_consensus.cc:482] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.295562 28257 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.201357 28049 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.292061 27708 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.324232 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.111857 28254 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.114203 28243 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.309996 25102 rpcz_store.cc:276] 0426 13:20:11.726932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.902444 (+175512us) service_pool.cc:225] Handling call
0426 13:20:11.909396 (+ 6952us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:11.965600 (+ 56204us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.138733 (+1173133us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b5400169c54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.139179 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127659776}
I20240426 13:20:13.342885 25105 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.115526 28096 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.351830 25620 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.326319 28257 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:13.200062 28124 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.414465 28221 log.cc:924] Time spent T 99a876ca7b6e4082b58359403a47730d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 2.066s user 0.000s sys 0.016s
W20240426 13:20:13.372288 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46436.1us
I20240426 13:20:13.415669 28124 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.376890 24950 raft_consensus.cc:1226] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20240426 13:20:13.435803 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1720) took 1712 ms. Trace:
I20240426 13:20:13.435914 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1841) took 2434 ms. Trace:
I20240426 13:20:13.436292 24950 rpcz_store.cc:276] 0426 13:20:11.723679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.328268 (+1604589us) service_pool.cc:225] Handling call
0426 13:20:13.376581 (+ 48313us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.435469 (+ 58888us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.435717 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.436358 25104 rpcz_store.cc:276] 0426 13:20:11.001299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.001441 (+ 142us) service_pool.cc:225] Handling call
0426 13:20:11.001704 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:11.929571 (+927867us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.435429 (+1505858us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54003e1cd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.435717 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119262976}
I20240426 13:20:13.437893 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.328003 28237 raft_consensus.cc:504] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.380496 28142 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.381181 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.382392 25096 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:13.382732 27916 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.388020 28173 raft_consensus.cc:482] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.389474 28174 consensus_queue.cc:1046] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240426 13:20:13.392710 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:13.393222 24952 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.017s sys 0.040s
I20240426 13:20:13.393153 28206 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817)
I20240426 13:20:13.393190 28126 leader_election.cc:288] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.328086 27451 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.395720 27923 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.396337 25618 raft_consensus.cc:2454] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 3.
I20240426 13:20:13.132382 24853 catalog_manager.cc:5484] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 9, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 9 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
I20240426 13:20:13.397733 28096 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:13.398223 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 50291.5us
I20240426 13:20:13.398854 28097 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.399444 25103 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:13.337020 27799 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.400012 28195 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:13.140674 27350 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.331281 28175 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.402037 28257 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.408816 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.409132 28248 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.409523 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.363236 28209 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.372898 28243 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.416981 28225 log.cc:924] Time spent T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 2.060s user 0.015s sys 0.004s
I20240426 13:20:13.416747 25588 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:20:13.426178 25099 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1855) took 1368 ms. Trace:
I20240426 13:20:13.427847 28159 raft_consensus.cc:482] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.430259 28025 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.431442 26679 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.432018 25022 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20240426 13:20:13.432489 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.438050 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.440639 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41286 (request call id 2091) took 1533 ms. Trace:
W20240426 13:20:13.441474 28155 consensus_meta.cc:287] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.570s user 0.005s sys 0.010s
I20240426 13:20:13.443096 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1858) took 1346 ms. Trace:
I20240426 13:20:13.299013 28161 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.444511 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 579) took 1254 ms. Trace:
I20240426 13:20:13.523150 24947 rpcz_store.cc:276] 0426 13:20:12.190490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.207756 (+ 17266us) service_pool.cc:225] Handling call
0426 13:20:13.444200 (+1236444us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540030b6d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.444368 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313253248}
I20240426 13:20:13.524443 28271 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.524720 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.449120 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 557) took 4428 ms. Trace:
I20240426 13:20:13.525303 28271 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.525738 24956 rpcz_store.cc:276] 0426 13:20:09.020709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:09.347614 (+326905us) service_pool.cc:225] Handling call
0426 13:20:12.018107 (+2670493us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.020675 (+ 2568us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.448659 (+1427984us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b5400366ad0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.448893 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6140787456}
I20240426 13:20:13.526783 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.527616 25624 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:13.528482 28286 raft_consensus.cc:482] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.528928 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.529291 28161 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.529095 28286 raft_consensus.cc:504] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } }
I20240426 13:20:13.529922 28155 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.530635 25030 raft_consensus.cc:2379] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 8: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:13.531402 28286 leader_election.cc:288] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.450249 25097 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1850) took 1757 ms. Trace:
W20240426 13:20:13.450932 27677 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240426 13:20:13.532099 28155 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.532897 28155 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.532927 28159 raft_consensus.cc:504] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.533655 28155 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.535073 28237 leader_election.cc:288] T bee343a4cf024dd09c3714d8cc509694 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.535248 28159 leader_election.cc:288] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.535653 28155 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.458273 25025 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:13.469995 25105 raft_consensus.cc:2396] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240426 13:20:13.536979 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58860 (request call id 1865) took 1074 ms. Trace:
I20240426 13:20:13.536861 26679 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.537372 25105 rpcz_store.cc:276] 0426 13:20:12.462849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.107072 (+644223us) service_pool.cc:225] Handling call
0426 13:20:13.536889 (+429817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.538252 27923 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.538516 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.538844 26679 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.518822 25022 pending_rounds.cc:76] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 3
I20240426 13:20:13.539868 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:13.478092 24952 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s user 0.018s sys 0.041s
I20240426 13:20:13.462463 28168 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.542338 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.543413 25617 raft_consensus.cc:2354] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:13.542938 28168 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.544106 25620 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 4.
I20240426 13:20:13.540200 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.545274 28168 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.503033 25103 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1861) took 1262 ms. Trace:
I20240426 13:20:13.545543 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:13.449349 27498 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 1.533s user 0.016s sys 0.003s
I20240426 13:20:13.516001 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.512900 24953 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:13.528203 28152 consensus_queue.cc:1046] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20240426 13:20:13.541836 28242 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20240426 13:20:13.541358 25100 rpcz_store.cc:276] 0426 13:20:12.097045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.217019 (+119974us) service_pool.cc:225] Handling call
0426 13:20:12.217287 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.104492 (+887205us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.442831 (+338339us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b540009c9d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.443047 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1983318784}
I20240426 13:20:13.468462 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1862) took 1199 ms. Trace:
I20240426 13:20:13.554407 25096 rpcz_store.cc:276] 0426 13:20:12.263918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.381519 (+1117601us) service_pool.cc:225] Handling call
0426 13:20:13.382032 (+ 513us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.463411 (+ 81379us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.463673 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.515942 25622 raft_consensus.cc:2379] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.527540 24956 raft_consensus.cc:1226] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20240426 13:20:13.514111 25099 rpcz_store.cc:276] 0426 13:20:12.058075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.190431 (+132356us) service_pool.cc:225] Handling call
0426 13:20:12.190692 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.380609 (+189917us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.425641 (+1045032us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b540001bdd4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.425972 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":435160320}
I20240426 13:20:13.519575 24955 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:13.546031 25103 rpcz_store.cc:276] 0426 13:20:12.235993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.363183 (+1127190us) service_pool.cc:225] Handling call
0426 13:20:13.399128 (+ 35945us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.498448 (+ 99320us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.498719 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.545291 27350 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:13.540158 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 123641us
I20240426 13:20:13.546875 25022 pending_rounds.cc:97] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20240426 13:20:13.558063 25022 raft_consensus.cc:2866] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:13.532199 25097 rpcz_store.cc:276] 0426 13:20:11.692789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.901192 (+208403us) service_pool.cc:225] Handling call
0426 13:20:11.901612 (+ 420us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.349717 (+448105us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.449769 (+1100052us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54000eae54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.450049 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1028578048}
I20240426 13:20:13.558809 25069 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:13.558925 28281 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.559058 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:13.559957 25097 raft_consensus.cc:595] T 3cd5654ae4294fcc8482e45f83032773 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:13.559713 28281 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.560413 25069 leader_election.cc:302] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:13.561525 25069 leader_election.cc:302] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:13.562273 24921 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:13.564265 28097 raft_consensus.cc:482] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.565014 28097 raft_consensus.cc:504] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.565604 28279 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.566339 28279 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.567602 28257 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.568445 28161 consensus_queue.cc:1046] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:13.569576 28152 raft_consensus.cc:3032] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.570137 28152 raft_consensus.cc:729] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.570973 27498 raft_consensus.cc:504] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:13.539798 28154 consensus_meta.cc:287] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.607s user 0.003s sys 0.008s
W20240426 13:20:13.552799 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 87001.1us
I20240426 13:20:13.549326 25104 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.560559 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1721) took 1829 ms. Trace:
I20240426 13:20:13.519150 24948 rpcz_store.cc:276] 0426 13:20:11.907048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.920206 (+ 13158us) service_pool.cc:225] Handling call
0426 13:20:13.440422 (+1520216us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000be854. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.440574 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2533716736}
I20240426 13:20:13.574169 24956 rpcz_store.cc:276] 0426 13:20:11.726553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.526790 (+1800237us) service_pool.cc:225] Handling call
0426 13:20:13.527227 (+ 437us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.555819 (+ 28592us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.556071 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.539449 28025 raft_consensus.cc:482] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.571435 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2089) took 1906 ms. Trace:
I20240426 13:20:13.532191 27451 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.540264 28268 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.527930 28271 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.550545 28055 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:13.572166 28097 leader_election.cc:288] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.575052 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.577618 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.511276 27799 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.580381 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.582199 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1718) took 1880 ms. Trace:
I20240426 13:20:13.587819 25618 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:13.591009 24951 rpcz_store.cc:276] 0426 13:20:11.698081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.972921 (+1274840us) service_pool.cc:225] Handling call
0426 13:20:12.974829 (+ 1908us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.072967 (+ 98138us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.073013 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.207495 (+134482us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.293051 (+ 85556us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.507313 (+214262us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.576987 (+ 69674us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.577040 (+ 53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.577068 (+ 28us) raft_consensus.cc:1671] finished
0426 13:20:13.577076 (+ 8us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.578573 (+ 1497us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91609,"prepare.queue_time_us":125036,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":65,"raft.queue_time_us":586876,"raft.run_cpu_time_us":974,"raft.run_wall_time_us":48374,"thread_start_us":237382,"threads_started":3,"wal-append.queue_time_us":208578}
I20240426 13:20:13.597151 28165 raft_consensus.cc:2781] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:13.597478 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.598109 28165 raft_consensus.cc:686] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:13.599099 24953 raft_consensus.cc:1226] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [7.4-9.5]
I20240426 13:20:13.599817 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1722) took 1869 ms. Trace:
I20240426 13:20:13.598268 25618 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.582096 28154 raft_consensus.cc:504] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.600288 24953 rpcz_store.cc:276] 0426 13:20:11.730808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.554407 (+1823599us) service_pool.cc:225] Handling call
0426 13:20:13.598692 (+ 44285us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:13.599546 (+ 854us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.599742 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.603387 25619 raft_consensus.cc:2379] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
W20240426 13:20:13.604945 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 95852.4us
I20240426 13:20:13.539449 28173 raft_consensus.cc:504] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.607609 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:13.571434 28152 consensus_queue.cc:260] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.518280 27708 raft_consensus.cc:504] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.587638 28055 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.607107 28025 raft_consensus.cc:504] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.614900 28025 leader_election.cc:288] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.616412 27946 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:13.617067 27946 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:13.617561 27946 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:13.619251 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.620805 24996 leader_election.cc:302] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:13.622169 28161 raft_consensus.cc:2726] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240426 13:20:13.623184 28161 consensus_queue.cc:1046] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:13.623095 24853 catalog_manager.cc:5484] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:13.624326 28257 raft_consensus.cc:3032] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.624604 27498 leader_election.cc:288] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.624742 28257 raft_consensus.cc:729] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.625020 24853 catalog_manager.cc:5484] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:13.625852 28257 consensus_queue.cc:260] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.627305 28257 raft_consensus.cc:3037] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.629448 27799 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.629810 25096 raft_consensus.cc:1226] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:13.578248 24952 rpcz_store.cc:276] 0426 13:20:11.664762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.105217 (+440455us) service_pool.cc:225] Handling call
0426 13:20:12.165316 (+ 60099us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.165575 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 5
0426 13:20:12.463783 (+298208us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.570934 (+1107151us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.571036 (+ 102us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.571109 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.571215 (+ 106us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.571365 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19821,"prepare.queue_time_us":1237492,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":20017,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":1337988,"threads_started":2,"wal-append.queue_time_us":872691}
I20240426 13:20:13.633176 24952 raft_consensus.cc:1226] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:13.633455 27410 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:13.623221 28243 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.637048 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1724) took 1904 ms. Trace:
I20240426 13:20:13.636907 28293 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:13.637717 28281 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.638010 24947 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:13.638279 27868 raft_consensus.cc:2781] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:13.629725 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.638918 27868 raft_consensus.cc:686] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:13.640082 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.638706 28154 leader_election.cc:288] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.640663 27350 consensus_queue.cc:1035] T 256415e42b6c444387115ab766a5a16c P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240426 13:20:13.649462 28248 consensus_queue.cc:1046] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:13.642375 25101 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:13.650632 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.651302 25102 raft_consensus.cc:2454] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:13.651578 25101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1860) took 1425 ms. Trace:
I20240426 13:20:13.645431 24947 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:13.651784 27708 leader_election.cc:288] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.654276 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1733) took 1919 ms. Trace:
I20240426 13:20:13.654693 24947 rpcz_store.cc:276] 0426 13:20:11.735219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.639540 (+1904321us) service_pool.cc:225] Handling call
0426 13:20:13.639925 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.654018 (+ 14093us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.654200 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.656217 25026 raft_consensus.cc:1226] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20240426 13:20:13.656752 24995 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:13.637507 24956 rpcz_store.cc:276] 0426 13:20:11.732935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.589054 (+1856119us) service_pool.cc:225] Handling call
0426 13:20:13.636421 (+ 47367us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.636722 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.636748 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.636808 (+ 60us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:13.636848 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.636893 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.637001 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.651086 28061 consensus_queue.cc:260] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.645417 25103 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:13.657022 25100 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 588) took 1370 ms. Trace:
I20240426 13:20:13.642431 25069 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:13.657616 25618 raft_consensus.cc:2454] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:13.627070 24853 catalog_manager.cc:5484] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } }
W20240426 13:20:13.659446 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 53621.5us
I20240426 13:20:13.657187 25101 rpcz_store.cc:276] 0426 13:20:12.226429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.360944 (+134515us) service_pool.cc:225] Handling call
0426 13:20:12.361349 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.650660 (+1289311us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.651345 (+ 685us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540001b3d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.651525 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2814869760}
I20240426 13:20:13.658016 25100 rpcz_store.cc:276] 0426 13:20:12.286467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.568174 (+1281707us) service_pool.cc:225] Handling call
0426 13:20:13.575347 (+ 7173us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.583754 (+ 8407us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.583789 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.583840 (+ 51us) raft_consensus.cc:1631] Marking committed up to 9
0426 13:20:13.583880 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.583943 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.656877 (+ 72934us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b54003ef8d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:13.656972 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15490304}
I20240426 13:20:13.633740 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1732) took 1898 ms. Trace:
I20240426 13:20:13.660790 25096 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:58852 (request call id 1163) took 1241 ms. Trace:
I20240426 13:20:13.661007 24952 rpcz_store.cc:276] 0426 13:20:11.734955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.632608 (+1897653us) service_pool.cc:225] Handling call
0426 13:20:13.632977 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.633515 (+ 538us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.633675 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.661406 25096 rpcz_store.cc:276] 0426 13:20:12.407657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.630748 (+1223091us) service_pool.cc:225] Handling call
0426 13:20:13.648799 (+ 18051us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.664968 25622 raft_consensus.cc:2362] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 3. Re-sending same reply.
I20240426 13:20:13.646474 28055 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.663048 28237 raft_consensus.cc:3032] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:13.697649 28237 raft_consensus.cc:729] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.699020 28237 consensus_queue.cc:260] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.684924 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2092) took 1777 ms. Trace:
I20240426 13:20:13.640107 27868 consensus_queue.cc:237] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.689713 28173 leader_election.cc:288] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.710551 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:13.711359 28281 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:13.711829 28281 raft_consensus.cc:482] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.712137 28188 leader_election.cc:288] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:13.691952 27869 consensus_meta.cc:287] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.588s user 0.005s sys 0.004s
I20240426 13:20:13.599658 28165 consensus_queue.cc:237] T 680b3a9e21d44d71987bc91ce797f547 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.714509 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.715312 25615 raft_consensus.cc:2454] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:13.653012 27943 raft_consensus.cc:3032] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:13.716187 27943 raft_consensus.cc:729] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.691380 28061 raft_consensus.cc:3037] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Advancing to term 5
W20240426 13:20:13.701359 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 112221us
I20240426 13:20:13.705929 24947 rpcz_store.cc:276] 0426 13:20:11.907462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.655518 (+1748056us) service_pool.cc:225] Handling call
0426 13:20:13.666128 (+ 10610us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.677722 (+ 11594us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.677761 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.684648 (+ 6887us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.684693 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.684750 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.684876 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.709539 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.712189 28281 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.696522 28152 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.716836 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.701609 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:13.721622 24996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 43581.9us
I20240426 13:20:13.712275 24948 raft_consensus.cc:1226] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:13.588953 28268 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:13.700156 28269 consensus_meta.cc:287] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.754s user 0.010s sys 0.001s
I20240426 13:20:13.711436 28279 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.765614 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1729) took 2031 ms. Trace:
I20240426 13:20:13.765982 24948 rpcz_store.cc:276] 0426 13:20:11.734178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.596580 (+1862402us) service_pool.cc:225] Handling call
0426 13:20:13.700498 (+103918us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.764350 (+ 63852us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.765549 (+ 1199us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240426 13:20:13.766976 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 121606us
I20240426 13:20:13.790685 25105 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:13.720503 27943 consensus_queue.cc:260] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.794744 27932 raft_consensus.cc:2781] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:13.795254 27932 raft_consensus.cc:482] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.795624 27932 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.799575 27677 consensus_queue.cc:1046] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:13.800173 28153 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:13.800478 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 217373us
W20240426 13:20:13.801034 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 298684us
I20240426 13:20:13.801427 27677 consensus_queue.cc:1046] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:13.802095 28142 raft_consensus.cc:3032] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:13.802372 27677 raft_consensus.cc:3032] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:13.802551 28142 raft_consensus.cc:729] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.802861 27677 raft_consensus.cc:729] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.804168 28295 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.804725 28295 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.805954 28195 consensus_queue.cc:1035] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:13.806273 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:13.806944 25618 raft_consensus.cc:3037] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 4 LEARNER]: Advancing to term 6
I20240426 13:20:13.807631 28034 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.789009 27350 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.800762 28243 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:13.724478 28055 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.831323 25615 raft_consensus.cc:2379] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.832082 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.805984 28062 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:13.836838 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.838102 25024 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.839653 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.847740 25623 raft_consensus.cc:2379] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:13.849046 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.850976 24947 raft_consensus.cc:1261] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240426 13:20:13.803807 28142 consensus_queue.cc:260] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.851653 24947 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2095) took 1943 ms. Trace:
I20240426 13:20:13.855170 24947 rpcz_store.cc:276] 0426 13:20:11.908380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.762381 (+1854001us) service_pool.cc:225] Handling call
0426 13:20:13.826980 (+ 64599us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.851400 (+ 24420us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.851595 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1443712}
I20240426 13:20:13.789631 28248 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20240426 13:20:13.854256 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 580) took 1663 ms. Trace:
I20240426 13:20:13.856283 24948 rpcz_store.cc:276] 0426 13:20:12.190785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.778051 (+1587266us) service_pool.cc:225] Handling call
0426 13:20:13.841247 (+ 63196us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.853838 (+ 12591us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.853877 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.853934 (+ 57us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:13.853979 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.854084 (+ 105us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.854198 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:13.858774 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1db66b7c1043e1bd43d7254a4dec62" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.859709 28055 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:13.859490 28281 raft_consensus.cc:504] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.861615 28281 leader_election.cc:288] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:13.865880 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2093) took 1958 ms. Trace:
W20240426 13:20:13.871120 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 46529.7us
I20240426 13:20:13.836344 28096 raft_consensus.cc:2726] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:13.837783 28062 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:13.876325 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.871608 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.866871 24956 rpcz_store.cc:276] 0426 13:20:11.907781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.663416 (+1755635us) service_pool.cc:225] Handling call
0426 13:20:13.689493 (+ 26077us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.689787 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 3
0426 13:20:13.690412 (+ 625us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.849320 (+158908us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.865052 (+ 15732us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.865326 (+ 274us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:13.865458 (+ 132us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.865663 (+ 205us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.865826 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23115,"prepare.queue_time_us":22607,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":137117,"threads_started":1}
I20240426 13:20:13.886632 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1731) took 2152 ms. Trace:
I20240426 13:20:13.800228 28161 consensus_queue.cc:1046] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.077s
I20240426 13:20:13.800834 28153 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.886760 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.891041 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.894258 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.898247 24947 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:13.898361 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.888980 24953 rpcz_store.cc:276] 0426 13:20:11.734687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.611353 (+1876666us) service_pool.cc:225] Handling call
0426 13:20:13.639498 (+ 28145us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.639853 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:13.658341 (+ 18488us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.886302 (+227961us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.886358 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.886401 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.886468 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.886583 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":217,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":36,"raft.queue_time_us":10590,"thread_start_us":195302,"threads_started":1}
I20240426 13:20:13.881160 28062 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
W20240426 13:20:13.896610 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54939.7us
I20240426 13:20:13.900460 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.900835 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.901043 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.904578 25025 raft_consensus.cc:2396] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240426 13:20:13.904958 25617 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:20:13.905323 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 139556us
I20240426 13:20:13.804033 27677 consensus_queue.cc:260] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:13.905784 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 76861us
I20240426 13:20:13.908579 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.909603 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.909852 27677 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:13.910306 25027 raft_consensus.cc:2454] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
W20240426 13:20:13.910698 28296 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240426 13:20:13.911303 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.911958 28239 consensus_queue.cc:1046] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20240426 13:20:13.913946 28141 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:13.914536 28141 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.916065 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1717) took 2142 ms. Trace:
I20240426 13:20:13.916301 24948 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.916538 24949 rpcz_store.cc:276] 0426 13:20:11.692632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.459636 (+767004us) service_pool.cc:225] Handling call
0426 13:20:12.459956 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:12.960067 (+500111us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:12.960163 (+ 96us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:12.961457 (+ 1294us) log.cc:842] Serialized 26 byte log entry
0426 13:20:12.993085 (+ 31628us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.300195 (+307110us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.834914 (+534719us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.834992 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.835043 (+ 51us) raft_consensus.cc:1671] finished
0426 13:20:13.835056 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.835203 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82425,"prepare.queue_time_us":393381,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":37767,"raft.queue_time_us":578371,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":23731,"thread_start_us":648904,"threads_started":3,"wal-append.queue_time_us":459918}
I20240426 13:20:13.918690 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.920378 25022 raft_consensus.cc:2454] T 6324309bd7cf40778c508e740dcf979d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.922307 24950 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1723) took 2189 ms. Trace:
I20240426 13:20:13.923425 28254 raft_consensus.cc:3032] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:13.924108 28153 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.924199 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.924960 25031 raft_consensus.cc:2454] T 499552c0877e46c5af41389ff1be3152 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:13.926429 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.927042 28239 consensus_queue.cc:1225] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240426 13:20:13.927614 28239 raft_consensus.cc:3032] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:13.927795 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:13.928056 28239 raft_consensus.cc:729] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.928480 25030 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 10
W20240426 13:20:13.929487 28265 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 1.596s user 0.004s sys 0.004s
I20240426 13:20:13.929076 28239 consensus_queue.cc:260] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.930580 28239 raft_consensus.cc:3037] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:13.871577 27350 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.897437 28237 raft_consensus.cc:3037] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:13.933279 27350 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:13.933933 28200 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:13.934468 28200 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:13.935952 25026 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47174 (request call id 434) took 1546 ms. Trace:
I20240426 13:20:13.936406 25026 rpcz_store.cc:276] 0426 13:20:12.389069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.389204 (+ 135us) service_pool.cc:225] Handling call
0426 13:20:12.424062 (+ 34858us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.666589 (+1242527us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.935661 (+269072us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000b1154. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.935878 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2832465280}
I20240426 13:20:13.922751 24950 rpcz_store.cc:276] 0426 13:20:11.732654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.567452 (+1834798us) service_pool.cc:225] Handling call
0426 13:20:13.568015 (+ 563us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.568479 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 4
0426 13:20:13.568819 (+ 340us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.724992 (+156173us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.791797 (+ 66805us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.791939 (+ 142us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.792205 (+ 266us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.792410 (+ 205us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.792478 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.922019 (+129541us) raft_consensus.cc:1671] finished
0426 13:20:13.922056 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.922230 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2327,"prepare.queue_time_us":6561,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":1796,"raft.queue_time_us":792,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":10351,"spinlock_wait_cycles":15683200,"thread_start_us":155692,"threads_started":1,"wal-append.queue_time_us":283695}
I20240426 13:20:13.938011 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.938355 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:13.938620 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.939558 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2094) took 2031 ms. Trace:
I20240426 13:20:13.939628 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.940398 25025 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 9.
I20240426 13:20:13.941419 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.942059 25025 raft_consensus.cc:2454] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:13.943476 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.944591 25029 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 11
I20240426 13:20:13.893110 27943 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:13.909400 27946 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.940058 24952 rpcz_store.cc:276] 0426 13:20:11.908083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.661794 (+1753711us) service_pool.cc:225] Handling call
0426 13:20:13.696526 (+ 34732us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:13.696953 (+ 427us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.696993 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:13.697595 (+ 602us) log.cc:842] Serialized 52 byte log entry
0426 13:20:13.822612 (+125017us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.848644 (+ 26032us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.848967 (+ 323us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.849046 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:13.939359 (+ 90313us) raft_consensus.cc:1671] finished
0426 13:20:13.939382 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.939508 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23904,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":8616,"raft.queue_time_us":6694,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":29863,"thread_start_us":124633,"threads_started":1,"wal-append.queue_time_us":239179}
I20240426 13:20:13.927635 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.938782 25026 raft_consensus.cc:2454] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:13.947638 25027 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:13.950896 24950 raft_consensus.cc:2354] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
I20240426 13:20:13.953002 25615 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:13.923827 28254 raft_consensus.cc:729] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:13.915508 27932 raft_consensus.cc:504] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:13.958426 25618 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEARNER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:13.961854 24954 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2090) took 2295 ms. Trace:
I20240426 13:20:13.957712 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.966202 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.976547 24995 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:13.979911 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:13.980644 25104 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:13.981086 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.981282 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.981184 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.982779 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.984252 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.984938 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:13.987731 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.990489 25031 raft_consensus.cc:2454] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:13.994107 24947 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:13.994217 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.993822 25026 raft_consensus.cc:2454] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:14.004442 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.005877 25023 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:13.982200 25069 connection.cc:232] RPC call timeout handler was delayed by -1.2178523119999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20240426 13:20:14.006057 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:13.997159 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.007448 25027 raft_consensus.cc:2396] T 08f19147e9214114a8749e2a8196bacd P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240426 13:20:14.020884 25028 raft_consensus.cc:2379] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 10: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:14.011185 24948 raft_consensus.cc:2454] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
W20240426 13:20:14.022089 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 64711.9us
I20240426 13:20:14.022630 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:13.990804 28299 log.cc:924] Time spent T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.195s user 0.001s sys 0.001s
I20240426 13:20:14.026819 25103 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 5.
I20240426 13:20:13.990546 28248 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:13.984958 28243 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:13.962404 24954 rpcz_store.cc:276] 0426 13:20:11.666851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.207002 (+540151us) service_pool.cc:225] Handling call
0426 13:20:12.207373 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:12.965727 (+758354us) raft_consensus.cc:1520] Early marking committed up to index 6
0426 13:20:13.009521 (+ 43794us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.346287 (+336766us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:13.346355 (+ 68us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:13.346412 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.961363 (+614951us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54003674d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:13.961424 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:13.961631 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45312,"prepare.queue_time_us":437686,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":231969,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":15127,"spinlock_wait_cycles":1742523136,"thread_start_us":324782,"threads_started":2,"wal-append.queue_time_us":412886}
I20240426 13:20:14.033341 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49912f064d045d8a1d8be2c08b97d44" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.031366 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.036732 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.037823 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:13.995586 28200 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.041922 25623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.042409 25617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.035754 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.046059 24953 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.046530 25623 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 11: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.043133 25617 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:14.045354 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.050560 24921 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.051750 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.045800 28141 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:14.057171 28310 log.cc:924] Time spent T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240426 13:20:14.061249 24995 leader_election.cc:398] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:14.061635 24995 leader_election.cc:401] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.061930 25029 raft_consensus.cc:2454] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.063238 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.064160 25026 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:14.065232 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.068588 25022 raft_consensus.cc:2396] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:14.070559 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.071446 25100 raft_consensus.cc:2379] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 3: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.071493 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.071915 25622 raft_consensus.cc:2454] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 6.
I20240426 13:20:14.072324 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.072096 28243 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.072496 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.073491 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.074193 24948 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 582) took 1835 ms. Trace:
I20240426 13:20:14.074302 28243 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.075268 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46260 (request call id 584) took 1120 ms. Trace:
I20240426 13:20:14.049942 28254 consensus_queue.cc:260] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.076072 27932 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.077764 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.078455 24952 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:20:14.079681 24952 raft_consensus.cc:1226] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.082501 24995 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.080780 27946 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.084717 28280 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.085225 28280 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.085289 25617 raft_consensus.cc:1226] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
I20240426 13:20:14.085683 28280 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.086009 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4680c139c46f44b49096bfd65f033a32" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.086869 24921 leader_election.cc:302] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.087312 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.087715 28281 raft_consensus.cc:2726] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:14.087971 25097 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:20:14.089131 24921 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.094960 25023 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.098182 25024 raft_consensus.cc:2396] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240426 13:20:14.075671 24955 rpcz_store.cc:276] 0426 13:20:12.954716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:12.970571 (+ 15855us) service_pool.cc:225] Handling call
0426 13:20:14.075120 (+1104549us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54002cb0d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda9031b43e 00007fda95259897 00007fda8e93dbd4 00007fda8e93db62 00007fda8e93da91 00007fda8e93da0c 00007fda8e93ccc2 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.075216 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1259656960}
I20240426 13:20:14.099530 24955 raft_consensus.cc:1226] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:14.031818 28248 raft_consensus.cc:482] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:14.100457 28248 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.100719 25589 leader_election.cc:302] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:14.101886 25030 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:14.102046 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.103361 28176 raft_consensus.cc:2781] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.103793 28176 raft_consensus.cc:482] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.104279 28176 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.074664 24948 rpcz_store.cc:276] 0426 13:20:12.238419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.073390 (+1834971us) service_pool.cc:225] Handling call
0426 13:20:14.073703 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.073889 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.073916 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:14.073958 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0426 13:20:14.073987 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.074039 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.074127 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.102945 28062 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.105942 28317 log.cc:924] Time spent T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240426 13:20:14.107023 28062 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.107614 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbdcc6d88d14934b64bc2a29382519f" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.108335 24950 raft_consensus.cc:3037] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.109015 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.116581 25103 raft_consensus.cc:2379] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.117652 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.118847 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.116597 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc419d5057b449c8d0888a41017ce2e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.089000 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.074549 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0238fe3803ea4654ae30109aa503e3a3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.121845 25025 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:14.122925 25589 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.124688 25029 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20240426 13:20:14.112442 25105 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 589) took 1685 ms. Trace:
I20240426 13:20:14.125365 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1737) took 1169 ms. Trace:
I20240426 13:20:14.112696 24955 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1739) took 1156 ms. Trace:
I20240426 13:20:14.125950 24952 rpcz_store.cc:276] 0426 13:20:12.955654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.079159 (+1123505us) service_pool.cc:225] Handling call
0426 13:20:14.079470 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.125032 (+ 45562us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.125289 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.126989 24955 rpcz_store.cc:276] 0426 13:20:12.956616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.098891 (+1142275us) service_pool.cc:225] Handling call
0426 13:20:14.099300 (+ 409us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.099893 (+ 593us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.112638 (+ 12745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240426 13:20:14.078758 25098 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240426 13:20:14.128132 25098 pending_rounds.cc:76] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting all ops after (but not including) 5
W20240426 13:20:14.112536 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82661.2us
I20240426 13:20:14.110873 28185 raft_consensus.cc:2726] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:14.128876 25098 pending_rounds.cc:97] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240426 13:20:14.109158 28153 raft_consensus.cc:2781] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:14.134086 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.124450 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.112455 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997ccc9a12742c3b149dc9747b2a0ae" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.128022 25069 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.110967 25621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 621) took 3801 ms. Trace:
I20240426 13:20:14.125607 25105 rpcz_store.cc:276] 0426 13:20:12.426695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.574753 (+1148058us) service_pool.cc:225] Handling call
0426 13:20:13.602510 (+ 27757us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.793631 (+191121us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.112221 (+318590us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54003e17d4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.112383 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":432132352}
I20240426 13:20:14.129277 25098 raft_consensus.cc:2866] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240426 13:20:14.139848 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 50800.9us
W20240426 13:20:14.139124 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 56699.5us
I20240426 13:20:14.142802 24995 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.143944 25097 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 5.
I20240426 13:20:14.145367 25097 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:14.148236 28174 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election won for term 11
I20240426 13:20:14.150311 25098 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55910 (request call id 586) took 2541 ms. Trace:
I20240426 13:20:14.150774 25098 rpcz_store.cc:276] 0426 13:20:11.608577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:11.608761 (+ 184us) service_pool.cc:225] Handling call
0426 13:20:11.643232 (+ 34471us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:14.149261 (+2506029us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.149999 (+ 738us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b540009ced4. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.150246 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5599939840}
I20240426 13:20:14.158046 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.156185 24950 raft_consensus.cc:2454] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:14.159293 25070 leader_election.cc:302] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.171689 24954 raft_consensus.cc:3037] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
W20240426 13:20:14.199762 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 46635.7us
W20240426 13:20:14.201133 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 58522.1us
I20240426 13:20:14.155543 24947 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.137877 28281 raft_consensus.cc:2781] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:14.196245 28061 raft_consensus.cc:2726] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240426 13:20:14.203194 27847 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20240426 13:20:14.203241 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 165436us
I20240426 13:20:14.208679 24921 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:14.212685 28303 log.cc:924] Time spent T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20240426 13:20:14.153649 25621 rpcz_store.cc:276] 0426 13:20:10.327743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:10.337603 (+ 9860us) service_pool.cc:225] Handling call
0426 13:20:10.413138 (+ 75535us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:10.413407 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 8
0426 13:20:10.746394 (+332987us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.939775 (+3193381us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b1800078460. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9628e4b1 00007fda9629b231 00007fda8d3ceb75 00007fda96208789 00007fda8d3ec0eb 00007fda8d3ea4ef 00007fda8d3e9985 00007fda9030f560 00007fda90321706 00007fda90332267 00007fda903321e8
0426 13:20:14.030134 (+ 90359us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0426 13:20:14.036361 (+ 6227us) raft_consensus.cc:1631] Marking committed up to 8
0426 13:20:14.110506 (+ 74145us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.110619 (+ 113us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.110791 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6391,"prepare.queue_time_us":438345,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":60281,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6962,"spinlock_wait_cycles":51846912,"thread_start_us":567882,"threads_started":3,"wal-append.queue_time_us":241973,"wal-append.run_cpu_time_us":1158,"wal-append.run_wall_time_us":1049308}
I20240426 13:20:14.214910 27911 raft_consensus.cc:482] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.216396 28242 raft_consensus.cc:2781] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.217823 25618 raft_consensus.cc:1226] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [term 5 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240426 13:20:14.218762 28257 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.219717 27677 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
W20240426 13:20:14.220204 28313 log.cc:924] Time spent T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240426 13:20:14.196452 27932 raft_consensus.cc:2726] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240426 13:20:14.221108 28255 raft_consensus.cc:2726] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11.
I20240426 13:20:14.221177 28268 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.221901 25030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.221959 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f533a7c13bae41539cdc0a425b282e53" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.229918 28281 raft_consensus.cc:482] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.223522 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.224216 28295 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.225454 25029 raft_consensus.cc:1261] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240426 13:20:14.226109 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.201799 25589 leader_election.cc:302] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.221902 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.254994 24953 raft_consensus.cc:2454] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:14.257447 24956 raft_consensus.cc:1226] T f49912f064d045d8a1d8be2c08b97d44 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240426 13:20:14.258725 24950 raft_consensus.cc:2454] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:20:14.261001 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 59295.5us
I20240426 13:20:14.261262 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.247289 28242 raft_consensus.cc:686] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Leader. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.234117 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.162432 28174 raft_consensus.cc:686] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [term 11 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.157951 28025 raft_consensus.cc:482] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.227056 24920 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.244454 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343363b25040c29f4c212b440dc2cc" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:14.240806 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 102189us
W20240426 13:20:14.246416 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 51023.4us
I20240426 13:20:14.222786 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708857f1bab4329a3f12081a3c1d669" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.264169 24995 leader_election.cc:302] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.268174 25023 raft_consensus.cc:2454] T 1708857f1bab4329a3f12081a3c1d669 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:14.261898 24953 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.262673 25589 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.254741 28281 raft_consensus.cc:3037] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.271847 24921 leader_election.cc:302] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.269989 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:14.267316 24947 raft_consensus.cc:2362] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10. Re-sending same reply.
I20240426 13:20:14.267199 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41288 (request call id 1730) took 2532 ms. Trace:
I20240426 13:20:14.262365 28062 raft_consensus.cc:2781] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.286768 25069 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.247280 27677 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.288674 24947 raft_consensus.cc:1261] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:14.268893 28295 raft_consensus.cc:2781] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.134747 28153 raft_consensus.cc:482] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:14.260625 27932 raft_consensus.cc:482] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.280627 24951 rpcz_store.cc:276] 0426 13:20:11.734418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.593428 (+1859010us) service_pool.cc:225] Handling call
0426 13:20:13.593974 (+ 546us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.660322 (+ 66348us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.660367 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.722962 (+ 62595us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.845537 (+122575us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:13.854611 (+ 9074us) log.cc:842] Serialized 14 byte log entry
0426 13:20:13.866316 (+ 11705us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.866535 (+ 219us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.266994 (+400459us) raft_consensus.cc:1671] finished
0426 13:20:14.267014 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.267146 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34608,"prepare.queue_time_us":4815,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":73803,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":17905,"spinlock_wait_cycles":3840,"thread_start_us":64452,"threads_started":1,"wal-append.queue_time_us":304661}
I20240426 13:20:14.247344 27911 raft_consensus.cc:504] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.286928 27847 raft_consensus.cc:2781] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:14.292161 24921 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.277302 28206 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.302150 28206 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.307736 28206 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.297633 25098 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240426 13:20:14.299149 25105 raft_consensus.cc:1226] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.299487 24921 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.295126 28311 raft_consensus.cc:2741] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240426 13:20:14.300827 27847 raft_consensus.cc:686] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [term 10 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.310272 25100 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:20:14.290685 28062 raft_consensus.cc:482] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.316372 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.295032 25589 leader_election.cc:302] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.264600 28176 raft_consensus.cc:504] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.291369 27350 raft_consensus.cc:2741] T be16e3d2b66a44c5ac4a072b891540a2 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240426 13:20:14.328984 25589 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:14.268527 28257 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.299669 28255 raft_consensus.cc:2781] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.298478 28061 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 FOLLOWER]: Leader election won for term 3
I20240426 13:20:14.327097 25022 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 10.
I20240426 13:20:14.332207 25589 leader_election.cc:302] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.287987 28248 raft_consensus.cc:504] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.271589 28025 raft_consensus.cc:504] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.337554 28227 raft_consensus.cc:2781] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:14.337690 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:14.338018 28227 raft_consensus.cc:482] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.338434 28227 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:14.338567 28248 leader_election.cc:288] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:14.339674 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 59617.4us
I20240426 13:20:14.337363 28190 consensus_queue.cc:1035] T dacba3e2733347ecb1304d307310c944 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240426 13:20:14.317838 28295 raft_consensus.cc:686] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.322367 28269 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:14.341507 25027 raft_consensus.cc:3037] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.338761 28171 leader_election.cc:288] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.342319 24921 leader_election.cc:302] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.356706 27847 consensus_queue.cc:237] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.358414 24921 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.359549 24921 leader_election.cc:302] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.324812 28062 raft_consensus.cc:3037] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.351928 28055 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.360702 28055 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.361012 28055 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 5 FOLLOWER]: Advancing to term 6
W20240426 13:20:14.348153 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 121355us
I20240426 13:20:14.362733 24920 leader_election.cc:302] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
W20240426 13:20:14.340903 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 76880.6us
I20240426 13:20:14.364217 24995 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.365557 24995 leader_election.cc:302] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.367316 24950 raft_consensus.cc:2396] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20240426 13:20:14.350975 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.368844 25620 raft_consensus.cc:2396] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20240426 13:20:14.368774 28295 consensus_queue.cc:237] T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.369971 28248 raft_consensus.cc:2726] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:14.367322 28239 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.353848 27869 consensus_queue.cc:1035] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:14.371871 28239 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.371948 24921 leader_election.cc:302] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.372488 28239 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.373071 28171 raft_consensus.cc:2726] T 242110e36ca34176b9d522087d440de6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:14.354876 28305 consensus_queue.cc:1046] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240426 13:20:14.355429 27410 consensus_queue.cc:1046] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240426 13:20:14.332005 28255 raft_consensus.cc:482] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.330410 28200 raft_consensus.cc:2726] T 4680c139c46f44b49096bfd65f033a32 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:14.354547 28061 raft_consensus.cc:686] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 3 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.367729 24995 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.374583 24953 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.356992 28025 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.348765 28316 raft_consensus.cc:2726] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:14.383765 24947 raft_consensus.cc:1261] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20240426 13:20:14.352356 28311 raft_consensus.cc:2726] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240426 13:20:14.300271 28153 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.390188 24953 raft_consensus.cc:1261] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20240426 13:20:14.380635 27350 raft_consensus.cc:2781] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.381485 28255 raft_consensus.cc:3037] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.335832 27932 raft_consensus.cc:504] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.395725 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.383097 28171 raft_consensus.cc:2726] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240426 13:20:14.382386 28279 raft_consensus.cc:2726] T 1708857f1bab4329a3f12081a3c1d669 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:14.401347 25098 raft_consensus.cc:1226] T 242110e36ca34176b9d522087d440de6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:14.403272 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.405886 24949 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2097) took 2206 ms. Trace:
I20240426 13:20:14.406320 24949 rpcz_store.cc:276] 0426 13:20:12.199279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.919623 (+1720344us) service_pool.cc:225] Handling call
0426 13:20:13.943829 (+ 24206us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.994445 (+ 50616us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.994477 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:14.036808 (+ 42331us) log.cc:842] Serialized 26 byte log entry
0426 13:20:14.254277 (+217469us) raft_consensus.cc:1631] Marking committed up to 4
0426 13:20:14.254334 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.254430 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.405676 (+151246us) raft_consensus.cc:1671] finished
0426 13:20:14.405704 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.405832 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":26649,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":24206,"thread_start_us":170738,"threads_started":1,"wal-append.queue_time_us":309378}
I20240426 13:20:14.379926 27869 consensus_queue.cc:1035] T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240426 13:20:14.407052 28055 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.409451 28061 consensus_queue.cc:237] T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.410818 28055 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.412222 27869 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.412936 27869 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.413739 28280 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.414975 24948 raft_consensus.cc:1226] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.415486 27869 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.391105 27923 raft_consensus.cc:3032] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:14.416293 27923 raft_consensus.cc:729] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.417232 27923 consensus_queue.cc:260] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.418586 27923 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.423686 28300 raft_consensus.cc:2741] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:14.377691 28174 consensus_queue.cc:237] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.426119 27677 leader_election.cc:288] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.395323 28200 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20240426 13:20:14.432242 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44388.4us
I20240426 13:20:14.432658 25069 leader_election.cc:302] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:14.433509 27869 raft_consensus.cc:2781] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:14.434182 27869 raft_consensus.cc:686] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.434409 25069 leader_election.cc:302] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:14.434475 28164 consensus_queue.cc:1035] T a6a69cdcf2ee4768a20888dd974c6ba7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:14.435544 25070 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.435266 27869 consensus_queue.cc:237] T 11fad756437243d2959f5998d6bf8309 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.436470 28268 raft_consensus.cc:2781] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:14.437057 28268 raft_consensus.cc:686] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:14.438036 28268 consensus_queue.cc:237] T be16e3d2b66a44c5ac4a072b891540a2 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.441155 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.441380 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.389000 28161 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20240426 13:20:14.448961 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 54053.9us
I20240426 13:20:14.387171 28269 raft_consensus.cc:482] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.443591 24954 raft_consensus.cc:2454] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:14.455600 24949 raft_consensus.cc:1261] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240426 13:20:14.392172 28025 raft_consensus.cc:2781] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:14.462056 25102 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.397197 27350 raft_consensus.cc:482] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
I20240426 13:20:14.450227 28161 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.461970 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.416010 28280 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.457249 27271 raft_consensus.cc:2741] T edc419d5057b449c8d0888a41017ce2e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240426 13:20:14.459986 28257 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.460798 28025 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.467309 27350 raft_consensus.cc:3037] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.466382 27946 raft_consensus.cc:2726] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:14.470371 28161 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.387537 28242 consensus_queue.cc:237] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.526288 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.534608 28141 consensus_queue.cc:1046] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:14.537254 25617 raft_consensus.cc:1261] T 4010c37c00d94e78b5e3260ad78e7904 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240426 13:20:14.537629 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.538327 25615 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 3.
I20240426 13:20:14.539331 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.539651 25622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.539964 25619 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:14.537292 25618 raft_consensus.cc:1261] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20240426 13:20:14.540740 25618 pending_rounds.cc:76] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting all ops after (but not including) 8
I20240426 13:20:14.541024 25618 pending_rounds.cc:97] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20240426 13:20:14.540949 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.541559 25618 raft_consensus.cc:2866] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:14.545539 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.508397 28025 raft_consensus.cc:3037] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:14.550096 27911 leader_election.cc:288] T 0238fe3803ea4654ae30109aa503e3a3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.551236 27810 raft_consensus.cc:3032] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:14.551630 27810 raft_consensus.cc:729] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.482520 28248 consensus_queue.cc:1046] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
W20240426 13:20:14.553011 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 73203.3us
I20240426 13:20:14.553704 27410 raft_consensus.cc:482] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.554201 25070 leader_election.cc:302] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.554343 27410 raft_consensus.cc:504] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.556618 27946 consensus_queue.cc:1035] T 08f19147e9214114a8749e2a8196bacd P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:14.555069 28145 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.558051 27946 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.559137 27410 leader_election.cc:288] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.558053 28145 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.561092 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:14.561910 27738 raft_consensus.cc:3032] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:14.562296 27738 raft_consensus.cc:729] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.540436 25622 raft_consensus.cc:2379] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 8: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.563299 27932 leader_election.cc:288] T dacba3e2733347ecb1304d307310c944 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.562312 28145 leader_election.cc:288] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.563228 27738 consensus_queue.cc:260] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.565136 24956 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46260 (request call id 581) took 2374 ms. Trace:
I20240426 13:20:14.565178 28096 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:14.565685 27738 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.566253 25104 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58860 (request call id 1864) took 2111 ms. Trace:
I20240426 13:20:14.565920 28096 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.568567 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:14.569249 25620 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 9: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:14.569806 28153 raft_consensus.cc:504] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.571708 24852 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39858 (request call id 288) took 36905 ms (client timeout 15000 ms). Trace:
W20240426 13:20:14.581540 28307 log.cc:924] Time spent T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.573s user 0.003s sys 0.000s
I20240426 13:20:14.575984 28254 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.566597 25104 rpcz_store.cc:276] 0426 13:20:12.454431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.579535 (+1125104us) service_pool.cc:225] Handling call
0426 13:20:13.579889 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0426 13:20:13.987361 (+407472us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.566044 (+578683us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400168d54. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda903180e1 00007fda90316b2e 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86
0426 13:20:14.566202 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921212672}
I20240426 13:20:14.586027 24948 raft_consensus.cc:1226] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 8. Current term is 10. Ops: []
I20240426 13:20:14.587100 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.587616 24948 raft_consensus.cc:3037] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.577628 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
W20240426 13:20:14.588122 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69606.3us
I20240426 13:20:14.573288 24853 catalog_manager.cc:5484] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 6, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:14.578511 28258 raft_consensus.cc:3032] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Stepping down as leader of term 9
I20240426 13:20:14.541519 24954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:14.579787 28190 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240426 13:20:14.565582 24956 rpcz_store.cc:276] 0426 13:20:12.191025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.880141 (+1689116us) service_pool.cc:225] Handling call
0426 13:20:14.077980 (+197839us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:14.257227 (+179247us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:14.259906 (+ 2679us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.564953 (+305047us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000785d0. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.565079 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":414414464}
I20240426 13:20:14.581283 28164 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:14.541893 25624 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.567957 28096 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.597612 25024 raft_consensus.cc:2454] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.602205 24954 raft_consensus.cc:2379] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:14.469792 27677 consensus_queue.cc:1035] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20240426 13:20:14.483803 28237 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20240426 13:20:14.554494 28142 raft_consensus.cc:3037] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.476851 28269 raft_consensus.cc:504] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.492254 28206 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.114s
I20240426 13:20:14.620455 24949 raft_consensus.cc:1261] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240426 13:20:14.507274 28243 consensus_queue.cc:1046] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20240426 13:20:14.627192 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.621748 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.601644 28258 raft_consensus.cc:729] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.628412 25069 leader_election.cc:398] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
I20240426 13:20:14.641197 24949 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20240426 13:20:14.641204 25069 leader_election.cc:401] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.643633 24950 raft_consensus.cc:2454] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 5.
I20240426 13:20:14.605566 28153 leader_election.cc:288] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.643179 24853 catalog_manager.cc:5484] T f4076b67a4fe4b3297158d094e10de08 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 5, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4). New cstate: current_term: 5 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:14.618541 28176 leader_election.cc:288] T a28ef3032ef44d2d89c750a8d4892f00 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.620927 27738 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:14.618130 28257 consensus_queue.cc:1046] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20240426 13:20:14.676851 24853 catalog_manager.cc:5484] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
W20240426 13:20:14.680302 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 308408us
I20240426 13:20:14.681196 24921 leader_election.cc:302] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.682291 24921 leader_election.cc:302] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.699957 24921 leader_election.cc:302] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.701026 24948 raft_consensus.cc:2454] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:14.644306 28034 consensus_queue.cc:1035] T bee343a4cf024dd09c3714d8cc509694 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20240426 13:20:14.709067 24921 leader_election.cc:302] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.649176 28281 raft_consensus.cc:504] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.711493 28284 raft_consensus.cc:3032] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:14.712452 28280 raft_consensus.cc:2781] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240426 13:20:14.674635 28190 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20240426 13:20:14.715026 28269 leader_election.cc:288] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.674607 28239 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.700189 28165 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240426 13:20:14.703359 28164 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.667198 27738 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.717260 24921 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.708946 28227 raft_consensus.cc:504] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.667377 28161 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.717664 24853 catalog_manager.cc:5484] T f04bd8996e6349879e25fca4e52dbda9 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 7, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:14.718770 28171 raft_consensus.cc:2781] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.729323 27677 raft_consensus.cc:3032] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Stepping down as leader of term 5
W20240426 13:20:14.734127 28323 log.cc:924] Time spent T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240426 13:20:14.734759 24956 raft_consensus.cc:1226] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: []
I20240426 13:20:14.735637 28281 leader_election.cc:288] T f04bd8996e6349879e25fca4e52dbda9 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.736099 28049 consensus_queue.cc:1046] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240426 13:20:14.708777 28096 raft_consensus.cc:2781] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240426 13:20:14.699589 27271 raft_consensus.cc:2781] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 13:20:14.581463 24852 rpcz_store.cc:269] 0426 13:19:37.665994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:19:37.678006 (+ 12012us) service_pool.cc:225] Handling call
0426 13:20:14.571477 (+36893471us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0426 13:20:05.923809 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0426 13:20:05.975846 (+ 52037us) write_op.cc:424] Acquiring schema lock in shared mode
0426 13:20:05.975901 (+ 55us) write_op.cc:427] Acquired schema lock
0426 13:20:05.975921 (+ 20us) tablet.cc:660] Decoding operations
0426 13:20:05.978050 (+ 2129us) write_op.cc:612] Acquiring the partition lock for write op
0426 13:20:05.978195 (+ 145us) write_op.cc:633] Partition lock acquired for write op
0426 13:20:05.978259 (+ 64us) tablet.cc:683] Acquiring locks for 10 operations
0426 13:20:06.016148 (+ 37889us) tablet.cc:699] Row locks acquired
0426 13:20:06.016211 (+ 63us) write_op.cc:260] PREPARE: finished
0426 13:20:06.016515 (+ 304us) write_op.cc:270] Start()
0426 13:20:06.016780 (+ 265us) write_op.cc:275] Timestamp: P: 1714137606016412 usec, L: 0
0426 13:20:06.016840 (+ 60us) op_driver.cc:331] REPLICATION: starting
0426 13:20:06.018049 (+ 1209us) log.cc:842] Serialized 4020 byte log entry
0426 13:20:08.112524 (+2094475us) op_driver.cc:444] REPLICATION: finished
0426 13:20:08.811709 (+699185us) write_op.cc:300] APPLY: starting
0426 13:20:08.811831 (+ 122us) tablet.cc:1365] starting BulkCheckPresence
0426 13:20:08.814769 (+ 2938us) tablet.cc:1368] finished BulkCheckPresence
0426 13:20:08.814902 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0426 13:20:08.816311 (+ 1409us) tablet.cc:1381] finished ApplyRowOperation cycle
0426 13:20:08.816651 (+ 340us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0426 13:20:08.816711 (+ 60us) write_op.cc:311] APPLY: finished
0426 13:20:08.817070 (+ 359us) log.cc:842] Serialized 77 byte log entry
0426 13:20:08.817251 (+ 181us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:08.817379 (+ 128us) write_op.cc:446] Released schema lock
0426 13:20:08.817444 (+ 65us) write_op.cc:340] FINISH: Updating metrics
0426 13:20:08.817806 (+ 362us) write_op.cc:481] Releasing partition, row and schema locks
0426 13:20:08.817835 (+ 29us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":446794,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":669790,"apply.run_cpu_time_us":3808,"apply.run_wall_time_us":6481,"num_ops":10,"prepare.queue_time_us":145590,"prepare.run_cpu_time_us":24379,"prepare.run_wall_time_us":375821,"raft.queue_time_us":1180886,"raft.run_cpu_time_us":15166,"raft.run_wall_time_us":469552,"replication_time_us":2090638,"thread_start_us":1501452,"threads_started":4,"wal-append.queue_time_us":464019,"wal-append.run_cpu_time_us":30051,"wal-append.run_wall_time_us":3326021}]]}
I20240426 13:20:14.776651 25027 raft_consensus.cc:2454] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
W20240426 13:20:14.780619 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 63436.3us
I20240426 13:20:14.721690 28284 raft_consensus.cc:729] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
W20240426 13:20:14.801725 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 446443us
W20240426 13:20:14.781865 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 419172us
I20240426 13:20:14.723984 28280 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.785902 24921 leader_election.cc:302] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.743939 27946 raft_consensus.cc:2726] T 5b1db66b7c1043e1bd43d7254a4dec62 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:14.758467 28316 raft_consensus.cc:3032] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:14.747946 25616 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44496 (request call id 627) took 3665 ms. Trace:
I20240426 13:20:14.783454 28279 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:14.779922 27677 raft_consensus.cc:729] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.783330 27271 raft_consensus.cc:482] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.781829 28096 raft_consensus.cc:482] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.827739 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.783236 28171 raft_consensus.cc:482] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.719065 28062 raft_consensus.cc:504] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.724334 28305 raft_consensus.cc:482] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:14.807184 24853 catalog_manager.cc:5484] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 2 to 5, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:14.806785 28280 raft_consensus.cc:3037] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:14.829034 24920 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters:
I20240426 13:20:14.815133 28281 raft_consensus.cc:2781] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240426 13:20:14.830418 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 44581.6us
I20240426 13:20:14.875577 24921 leader_election.cc:302] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.831022 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.831095 28161 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.832141 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:14.833012 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.876823 25102 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.877019 25098 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:14.836503 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.836984 27738 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:14.837066 28284 consensus_queue.cc:260] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.816123 28279 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.838258 28239 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.839143 28227 leader_election.cc:288] T b9910a5336aa4e3cab11dca4a556fabf P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:14.839697 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.907095 25099 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.840364 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.841434 27677 consensus_queue.cc:260] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.746434 27810 consensus_queue.cc:260] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.813334 25616 rpcz_store.cc:276] 0426 13:20:11.082553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:13.428558 (+2346005us) service_pool.cc:225] Handling call
0426 13:20:13.499921 (+ 71363us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:13.540363 (+ 40442us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:13.540402 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:13.541142 (+ 740us) log.cc:842] Serialized 26 byte log entry
0426 13:20:13.855073 (+313931us) raft_consensus.cc:1631] Marking committed up to 5
0426 13:20:13.869506 (+ 14433us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:13.873580 (+ 4074us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.298284 (+424704us) raft_consensus.cc:1671] finished
0426 13:20:14.298305 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.747540 (+449235us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b5400473950. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda9029c5f1 00007fda90316b99 00007fda95258f59 00007fda8e93b654 00007fda8e93b5e2 00007fda8e93b511 00007fda8e93b48c 00007fda8e93a742 00007fda8e73ea4c 00007fda8e73de86 00007fda8e73d82f
0426 13:20:14.747768 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113987968,"thread_start_us":288030,"threads_started":1,"wal-append.queue_time_us":526244}
I20240426 13:20:14.827591 27271 raft_consensus.cc:3037] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:14.843881 28025 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.816473 28316 raft_consensus.cc:729] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.804632 27350 raft_consensus.cc:504] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.848155 28096 raft_consensus.cc:3037] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.850764 27932 raft_consensus.cc:3032] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Stepping down as leader of term 3
I20240426 13:20:14.856518 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.857828 25589 leader_election.cc:302] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.858747 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:14.926602 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68850.9us
I20240426 13:20:14.859513 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:14.860890 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:14.927608 27738 raft_consensus.cc:2781] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:14.927983 25029 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 7.
I20240426 13:20:14.928284 27738 raft_consensus.cc:686] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:14.861937 24952 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2101) took 1063 ms. Trace:
I20240426 13:20:14.929231 27738 consensus_queue.cc:237] T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:14.929354 24952 rpcz_store.cc:276] 0426 13:20:13.798926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.140923 (+341997us) service_pool.cc:225] Handling call
0426 13:20:14.197307 (+ 56384us) raft_consensus.cc:1457] Updating replica for 1 ops
0426 13:20:14.197789 (+ 482us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.197831 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0426 13:20:14.216153 (+ 18322us) log.cc:842] Serialized 26 byte log entry
0426 13:20:14.453298 (+237145us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:14.453333 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.453389 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:14.861725 (+408336us) raft_consensus.cc:1671] finished
0426 13:20:14.861749 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:14.861887 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":968,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":2443,"thread_start_us":236424,"threads_started":1,"wal-append.queue_time_us":420592}
I20240426 13:20:14.721120 28258 consensus_queue.cc:260] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.931622 28258 raft_consensus.cc:3037] T 81d7dd33a3494a2d94627156d2fc1615 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:14.932963 27677 raft_consensus.cc:3037] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:14.828445 25031 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 2.
W20240426 13:20:14.872112 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 325663us
I20240426 13:20:14.873710 25027 raft_consensus.cc:1226] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20240426 13:20:14.874920 25096 raft_consensus.cc:1226] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:14.937451 24995 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
W20240426 13:20:14.897274 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 68315.1us
I20240426 13:20:14.938457 24995 leader_election.cc:302] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.902887 24921 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.872383 28171 raft_consensus.cc:3037] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.911816 25102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.913756 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:14.888607 28281 raft_consensus.cc:482] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.904786 28279 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:14.915416 24853 catalog_manager.cc:5484] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:14.905292 28269 raft_consensus.cc:2781] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:14.924163 28316 consensus_queue.cc:260] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.905193 28295 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.020s sys 0.039s
I20240426 13:20:14.830068 28255 raft_consensus.cc:504] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:14.904271 28330 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:14.926434 25022 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 11.
I20240426 13:20:14.941557 24995 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46106ff992984d33bb4a4d8f30162639; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.941828 25589 leader_election.cc:302] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
W20240426 13:20:14.947427 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 300580us
I20240426 13:20:14.952390 24995 leader_election.cc:398] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.952870 24995 leader_election.cc:401] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240426 13:20:14.925573 28284 raft_consensus.cc:3037] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:14.896534 28305 raft_consensus.cc:504] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:14.957429 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 54619.2us
I20240426 13:20:14.960299 24921 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:14.926050 27932 raft_consensus.cc:729] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:14.945104 28239 consensus_queue.cc:1035] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20240426 13:20:15.039474 25624 raft_consensus.cc:1226] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 10. Current term is 11. Ops: []
W20240426 13:20:15.010818 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 386649us
I20240426 13:20:14.943887 28257 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240426 13:20:14.954205 28269 raft_consensus.cc:482] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:14.969623 28237 raft_consensus.cc:2726] T 59632e528afa44ca98a626355e1e4660 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 59632e528afa44ca98a626355e1e4660 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:14.954825 28206 consensus_queue.cc:1035] T e915e90a1a6847899575125ef7e68f2a P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
W20240426 13:20:14.958190 28295 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.020s sys 0.040s
W20240426 13:20:14.993327 24920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 54409.9us
W20240426 13:20:15.036319 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 77831.1us
I20240426 13:20:14.945564 27943 raft_consensus.cc:2781] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240426 13:20:14.968250 28243 raft_consensus.cc:2726] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240426 13:20:15.010026 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 49803.9us
I20240426 13:20:14.954367 28330 raft_consensus.cc:482] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:14.986785 28325 raft_consensus.cc:2781] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20240426 13:20:15.009595 24853 catalog_manager.cc:5484] T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } }
I20240426 13:20:14.986342 28326 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.009737 25589 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.017731 28062 leader_election.cc:288] T 4010c37c00d94e78b5e3260ad78e7904 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:14.944398 28281 raft_consensus.cc:3037] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.043174 25069 leader_election.cc:302] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.056308 24951 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2108) took 1130 ms. Trace:
I20240426 13:20:15.056564 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e915e90a1a6847899575125ef7e68f2a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.062970 24951 rpcz_store.cc:276] 0426 13:20:13.925713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.306136 (+380423us) service_pool.cc:225] Handling call
0426 13:20:14.306671 (+ 535us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:14.307189 (+ 518us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.307230 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:14.309099 (+ 1869us) log.cc:842] Serialized 52 byte log entry
0426 13:20:14.801718 (+492619us) raft_consensus.cc:1631] Marking committed up to 3
0426 13:20:14.831022 (+ 29304us) log.cc:842] Serialized 14 byte log entry
0426 13:20:14.831230 (+ 208us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.831296 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:15.056098 (+224802us) raft_consensus.cc:1671] finished
0426 13:20:15.056118 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:15.056254 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29541,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":631,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":152,"thread_start_us":413021,"threads_started":1,"wal-append.queue_time_us":673476}
I20240426 13:20:14.986519 27810 raft_consensus.cc:3037] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.050798 25070 leader_election.cc:302] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:14.970865 28164 raft_consensus.cc:2726] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817) with higher term. Message: Invalid argument: T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10.
W20240426 13:20:15.057817 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 57478.5us
I20240426 13:20:15.092015 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:14.989413 28316 raft_consensus.cc:3037] T f49912f064d045d8a1d8be2c08b97d44 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.058477 24853 catalog_manager.cc:5484] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:15.056620 28269 raft_consensus.cc:3037] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.047545 28200 raft_consensus.cc:482] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.047340 28328 raft_consensus.cc:2781] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240426 13:20:15.054658 27943 raft_consensus.cc:482] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.056371 28257 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1590f80bdade41c6a755b8501c55fe16)
W20240426 13:20:15.111850 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 61108.7us
W20240426 13:20:15.113037 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 69945.8us
I20240426 13:20:15.057996 28055 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240426 13:20:15.058607 28243 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.056589 28325 raft_consensus.cc:482] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.114004 28305 leader_election.cc:288] T 256415e42b6c444387115ab766a5a16c P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:15.120111 28255 leader_election.cc:288] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.120280 28337 log.cc:924] Time spent T b9910a5336aa4e3cab11dca4a556fabf P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20240426 13:20:15.056275 28330 raft_consensus.cc:3037] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.094189 28062 raft_consensus.cc:2726] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240426 13:20:15.064603 28237 raft_consensus.cc:482] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.089428 28152 raft_consensus.cc:482] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.122537 25069 leader_election.cc:302] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.126677 25589 leader_election.cc:302] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:15.056041 27271 raft_consensus.cc:504] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.064774 28254 consensus_queue.cc:1046] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20240426 13:20:15.096653 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.141510 24853 catalog_manager.cc:5484] T f533a7c13bae41539cdc0a425b282e53 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.114820 28328 raft_consensus.cc:482] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:15.148464 25102 raft_consensus.cc:3032] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Stepping down as leader of term 7
I20240426 13:20:15.122404 27943 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.121621 28257 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:15.144580 25589 leader_election.cc:302] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.143790 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
I20240426 13:20:15.128795 27738 raft_consensus.cc:2781] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.121274 28142 consensus_queue.cc:1046] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20240426 13:20:15.144652 24853 catalog_manager.cc:5484] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 6, leader changed from 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 6 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:15.145644 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.170584 25620 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 8.
I20240426 13:20:15.126540 28025 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 4 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.130272 27350 leader_election.cc:288] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.134655 28243 raft_consensus.cc:504] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.048383 27932 consensus_queue.cc:260] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.175159 27738 raft_consensus.cc:482] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1e87e7bb414642aeb72fe0d2ff72cc2e)
I20240426 13:20:15.214421 28295 raft_consensus.cc:2726] T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.230010 27738 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.230657 28280 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.152132 28055 raft_consensus.cc:2781] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240426 13:20:15.232528 28055 raft_consensus.cc:482] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.232892 28055 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:15.229394 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 98478.6us
I20240426 13:20:15.235080 27271 leader_election.cc:288] T 6324309bd7cf40778c508e740dcf979d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.236761 27350 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.141157 28339 raft_consensus.cc:2726] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.239601 27932 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
W20240426 13:20:15.243218 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 96245.9us
I20240426 13:20:15.244484 25070 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
W20240426 13:20:15.249171 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 126690us
I20240426 13:20:15.250274 28268 raft_consensus.cc:2781] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:15.250771 28268 raft_consensus.cc:686] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.251258 25588 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.251628 28268 consensus_queue.cc:237] T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.251897 28227 consensus_queue.cc:1035] T 0238fe3803ea4654ae30109aa503e3a3 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240426 13:20:15.254515 28344 consensus_queue.cc:1035] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240426 13:20:15.256328 25069 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:15.257205 27916 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.258325 25069 leader_election.cc:302] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.148952 25102 raft_consensus.cc:729] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.117566 28200 raft_consensus.cc:504] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.260149 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.260797 28346 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20240426 13:20:15.261447 28165 raft_consensus.cc:2781] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:15.262032 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896b58b7766401eac0a7f69574827f1" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.263676 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.264058 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ec72789da4a7383ca5ce0f90a4563" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.264676 25030 raft_consensus.cc:1226] T 9911e0422347463c8781b2574c023744 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20240426 13:20:15.264959 25025 raft_consensus.cc:1226] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:15.265271 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.265472 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242110e36ca34176b9d522087d440de6" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.258384 28243 leader_election.cc:288] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.266554 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:15.269209 28345 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
W20240426 13:20:15.212658 25589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 68345.4us
I20240426 13:20:15.237390 27350 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: NON_VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: true } }
I20240426 13:20:15.278784 28326 raft_consensus.cc:2726] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240426 13:20:15.243759 28049 raft_consensus.cc:482] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:15.213438 28343 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20240426 13:20:15.220743 28329 log.cc:924] Time spent T edc419d5057b449c8d0888a41017ce2e P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20240426 13:20:15.289171 28330 raft_consensus.cc:504] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.364794 27350 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.363940 28049 raft_consensus.cc:504] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.368969 28049 leader_election.cc:288] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.369766 28338 log.cc:924] Time spent T dacba3e2733347ecb1304d307310c944 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Append to log took a long time: real 0.237s user 0.002s sys 0.002s
I20240426 13:20:15.370492 25024 raft_consensus.cc:2454] T c896b58b7766401eac0a7f69574827f1 P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 11.
W20240426 13:20:15.296281 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 240138us
I20240426 13:20:15.371168 24953 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41286 (request call id 2112) took 1444 ms. Trace:
I20240426 13:20:15.371469 28330 leader_election.cc:288] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.131763 28325 raft_consensus.cc:3037] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 11 FOLLOWER]: Advancing to term 12
I20240426 13:20:15.292446 25096 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:15.297847 25099 raft_consensus.cc:2454] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 6.
I20240426 13:20:15.370085 27943 raft_consensus.cc:504] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:15.177155 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.4:40069, user_credentials={real_user=slave}} blocked reactor thread for 75613.4us
I20240426 13:20:15.362920 25022 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:15.375833 24996 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:15.383966 24995 leader_election.cc:302] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.151582 28328 raft_consensus.cc:3037] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.385342 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.385852 24921 leader_election.cc:302] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.386323 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.386618 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.387418 24921 leader_election.cc:302] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.388366 25101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f413a75d3c249dbb47c1ec61978c2ea" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.388505 24853 catalog_manager.cc:5484] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 3 to 4, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 4 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.390038 24853 catalog_manager.cc:5484] T eb99666c521d49ada45165233068daf3 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.377961 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.392330 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.393119 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.393666 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.394145 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.394829 25100 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 1590f80bdade41c6a755b8501c55fe16 in this term.
I20240426 13:20:15.387428 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.399649 28330 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.143993 28237 raft_consensus.cc:504] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.371523 24953 rpcz_store.cc:276] 0426 13:20:13.926900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0426 13:20:14.393260 (+466360us) service_pool.cc:225] Handling call
0426 13:20:14.583354 (+190094us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b5800001598. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d3b5071 00007fda8d3b4f22 00007fda962a76d9 00007fda951e4270 00007fda952aa27b 00007fda952aa301 00007fda9526df20 00007fda95259469 00007fda95258f12 00007fda8e93b654
0426 13:20:14.583554 (+ 200us) raft_consensus.cc:1457] Updating replica for 2 ops
0426 13:20:14.584014 (+ 460us) raft_consensus.cc:1520] Early marking committed up to index 0
0426 13:20:14.584059 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0426 13:20:14.584618 (+ 559us) log.cc:842] Serialized 52 byte log entry
0426 13:20:14.811990 (+227372us) raft_consensus.cc:1631] Marking committed up to 2
0426 13:20:14.812056 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0426 13:20:14.812122 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0426 13:20:15.251044 (+438922us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b44000232cc. stack: 00007fda8d3ba863 00007fda8d3ba760 00007fda8cd1f4e4 00007fda96a34924 00007fda96a348b9 00007fda8d233811 00007fda8d301345 00007fda8d301d82 00007fda8ffaa391 00007fda8ffa5c9a 00007fda8ff99a6e 00007fda8ff98db6 00007fda8ff989d8 00007fda8ffa23d1 00007fda8ffa2389 00007fda8ffa2319
0426 13:20:15.370965 (+119921us) raft_consensus.cc:1671] finished
0426 13:20:15.370986 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0426 13:20:15.371122 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17875,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8976,"spinlock_wait_cycles":99176960,"thread_start_us":226995,"threads_started":1,"wal-append.queue_time_us":457700}
I20240426 13:20:15.249328 27738 raft_consensus.cc:504] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:15.383098 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 124832us
I20240426 13:20:15.414503 25069 leader_election.cc:302] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16
I20240426 13:20:15.144708 28152 raft_consensus.cc:504] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.400894 24921 leader_election.cc:302] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.417822 24921 leader_election.cc:302] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.353900 25102 consensus_queue.cc:260] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.391276 24853 catalog_manager.cc:5484] T 5b1db66b7c1043e1bd43d7254a4dec62 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 5, leader changed from 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1) to 46106ff992984d33bb4a4d8f30162639 (127.23.61.2). New cstate: current_term: 5 leader_uuid: "46106ff992984d33bb4a4d8f30162639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } health_report { overall_health: UNKNOWN } } }
I20240426 13:20:15.387697 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.371654 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.420384 25024 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:15.408627 25589 leader_election.cc:302] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:15.151196 25621 raft_consensus.cc:3037] T 7f413a75d3c249dbb47c1ec61978c2ea P 1590f80bdade41c6a755b8501c55fe16 [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:15.150930 28254 consensus_queue.cc:1225] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Peer 30c2bb0adfa54bbfa3cb39773d4043c8 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240426 13:20:15.363896 28326 raft_consensus.cc:482] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.422739 25028 raft_consensus.cc:1226] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:15.423466 25025 raft_consensus.cc:1226] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 6. Current term is 7. Ops: []
I20240426 13:20:15.423319 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.423729 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.405943 25101 raft_consensus.cc:2454] T 7f413a75d3c249dbb47c1ec61978c2ea P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:15.427482 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.288017 28257 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.381642 28305 raft_consensus.cc:2726] T a2212db2dbcf486888aefb7dd0174287 P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:15.426230 24853 catalog_manager.cc:5484] T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 1 to 2, leader changed from 46106ff992984d33bb4a4d8f30162639 (127.23.61.2) to 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3). New cstate: current_term: 2 leader_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } }
I20240426 13:20:15.440361 25027 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 8.
I20240426 13:20:15.269219 28269 raft_consensus.cc:504] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.410321 27350 raft_consensus.cc:482] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.443926 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.446348 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.450695 25097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551154b9c3494e4da288affc4b88c361"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:15.451419 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718dfae59b1148c2b6cc522821ea4ea4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.451520 25097 raft_consensus.cc:595] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:15.452445 24952 raft_consensus.cc:2396] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20240426 13:20:15.452415 25097 raft_consensus.cc:603] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:15.417907 28165 raft_consensus.cc:686] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.426718 28316 raft_consensus.cc:2781] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Leader election won for term 6
I20240426 13:20:15.416127 28025 raft_consensus.cc:482] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 46106ff992984d33bb4a4d8f30162639)
I20240426 13:20:15.455698 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f19147e9214114a8749e2a8196bacd" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.456993 25022 raft_consensus.cc:1226] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 8. Current term is 9. Ops: []
I20240426 13:20:15.456298 24853 catalog_manager.cc:5484] T 4680c139c46f44b49096bfd65f033a32 P 46106ff992984d33bb4a4d8f30162639 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1590f80bdade41c6a755b8501c55fe16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } } }
I20240426 13:20:15.463428 25025 raft_consensus.cc:1226] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [term 8 LEADER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 7. Current term is 8. Ops: []
I20240426 13:20:15.391870 28328 raft_consensus.cc:504] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.464637 24948 raft_consensus.cc:1261] T f7408112bace482693c999f678aa9d73 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240426 13:20:15.467096 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.433768 27810 raft_consensus.cc:2781] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.400156 28330 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.440209 28164 raft_consensus.cc:2781] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader election won for term 8
I20240426 13:20:15.464927 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee343a4cf024dd09c3714d8cc509694" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.479437 24956 raft_consensus.cc:2454] T bee343a4cf024dd09c3714d8cc509694 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:15.483379 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.488754 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.468600 28034 consensus_queue.cc:1035] T eb99666c521d49ada45165233068daf3 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240426 13:20:15.490928 28340 raft_consensus.cc:2781] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader election won for term 9
I20240426 13:20:15.494055 28340 raft_consensus.cc:686] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.495167 28049 raft_consensus.cc:3032] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Stepping down as leader of term 10
I20240426 13:20:15.495064 28340 consensus_queue.cc:237] T 3fe8f3015c01480fb67b39d3f5a947c7 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.429731 28326 raft_consensus.cc:504] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.497648 28326 leader_election.cc:288] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.498025 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4010c37c00d94e78b5e3260ad78e7904" candidate_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.498243 28326 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240426 13:20:15.495616 28049 raft_consensus.cc:729] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:15.499701 25027 raft_consensus.cc:1226] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Rejecting Update request from peer 30c2bb0adfa54bbfa3cb39773d4043c8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:15.501128 25070 leader_election.cc:302] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8; no voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.499609 28049 consensus_queue.cc:260] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.502144 28152 leader_election.cc:288] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.502487 27911 raft_consensus.cc:2726] T 8a2ec72789da4a7383ca5ce0f90a4563 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240426 13:20:15.502745 28152 raft_consensus.cc:482] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.503387 25070 leader_election.cc:302] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:15.504135 28049 raft_consensus.cc:3037] T 8a2ec72789da4a7383ca5ce0f90a4563 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:15.503233 27911 consensus_queue.cc:1046] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240426 13:20:15.505367 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.512414 24950 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.515959 28339 consensus_queue.cc:1046] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240426 13:20:15.520287 28330 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.520504 28339 raft_consensus.cc:3032] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Stepping down as leader of term 5
I20240426 13:20:15.520870 28330 raft_consensus.cc:3037] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.521077 28339 raft_consensus.cc:729] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.522511 28062 raft_consensus.cc:2781] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240426 13:20:15.523058 28062 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.523474 28062 raft_consensus.cc:3037] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 FOLLOWER]: Advancing to term 9
I20240426 13:20:15.448547 27498 consensus_queue.cc:1035] T 1708857f1bab4329a3f12081a3c1d669 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.063s
I20240426 13:20:15.524427 28165 consensus_queue.cc:237] T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.520313 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.426486 28171 raft_consensus.cc:504] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.536059 24956 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:15.501168 28096 consensus_meta.cc:287] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.577s user 0.009s sys 0.001s
I20240426 13:20:15.478592 27810 raft_consensus.cc:482] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.458745 28061 consensus_queue.cc:1035] T 0d8fda82f95348219809212d99d6704e P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20240426 13:20:15.511319 28316 raft_consensus.cc:686] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEADER]: Becoming Leader. State: Replica: 1590f80bdade41c6a755b8501c55fe16, State: Running, Role: LEADER
I20240426 13:20:15.562590 24947 raft_consensus.cc:1226] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20240426 13:20:15.569471 27271 raft_consensus.cc:2726] T 718dfae59b1148c2b6cc522821ea4ea4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.572590 27943 leader_election.cc:288] T d066b73b2e464d80b7b667e05602d879 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 election: Requested vote from peers 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.572919 25027 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:20:15.573500 27911 consensus_queue.cc:1046] T 2d6f5c5283464089ad1bc18c13b427cc P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240426 13:20:15.575232 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.576099 28200 leader_election.cc:288] T 3014c46944bc4cbb82624ac909221634 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.471040 27350 raft_consensus.cc:504] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.535559 28164 raft_consensus.cc:686] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Leader. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.576653 28255 raft_consensus.cc:3032] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:15.475407 28025 raft_consensus.cc:504] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.579314 25026 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 7.
I20240426 13:20:15.503283 28152 raft_consensus.cc:504] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.586704 28330 raft_consensus.cc:2726] T f533a7c13bae41539cdc0a425b282e53 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240426 13:20:15.547556 27810 raft_consensus.cc:3037] T 3cd5654ae4294fcc8482e45f83032773 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.661027 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.664050 25031 raft_consensus.cc:3037] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.662638 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.601454 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4cb6239dd94f7693fdaacdc8ae8a8f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.665966 24956 raft_consensus.cc:3037] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Advancing to term 10
I20240426 13:20:15.659878 24949 raft_consensus.cc:2454] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 10.
I20240426 13:20:15.663735 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.651600 28279 consensus_meta.cc:287] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.679s user 0.010s sys 0.001s
I20240426 13:20:15.598544 28190 consensus_queue.cc:1046] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20240426 13:20:15.659647 27350 leader_election.cc:288] T 0d8fda82f95348219809212d99d6704e P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.572597 28326 raft_consensus.cc:2726] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240426 13:20:15.619848 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.619992 25101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd"
dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
mode: GRACEFUL
new_leader_uuid: "46106ff992984d33bb4a4d8f30162639"
from {username='slave'} at 127.0.0.1:49046
I20240426 13:20:15.621261 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
W20240426 13:20:15.585633 28349 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
W20240426 13:20:15.580857 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 90351.3us
I20240426 13:20:15.583031 28034 raft_consensus.cc:3032] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:15.539520 28280 leader_election.cc:288] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
W20240426 13:20:15.651460 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 233685us
I20240426 13:20:15.707432 24921 leader_election.cc:302] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:15.652365 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59632e528afa44ca98a626355e1e4660" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.616479 28350 raft_consensus.cc:2726] T 0238fe3803ea4654ae30109aa503e3a3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240426 13:20:15.621150 27847 consensus_queue.cc:1035] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20240426 13:20:15.625705 28255 raft_consensus.cc:729] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 46106ff992984d33bb4a4d8f30162639, State: Running, Role: LEADER
I20240426 13:20:15.654495 28328 leader_election.cc:288] T aca63e93aec8413b83ec63e853c8f403 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 6 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.654629 28269 leader_election.cc:288] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.655084 28257 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:15.648725 27738 leader_election.cc:288] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:15.655779 28171 leader_election.cc:288] T eb99666c521d49ada45165233068daf3 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.546609 28096 raft_consensus.cc:504] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.616704 27498 consensus_queue.cc:1046] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240426 13:20:15.657104 28166 consensus_queue.cc:1046] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20240426 13:20:15.768000 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacba3e2733347ecb1304d307310c944" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.718508 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.657984 27923 raft_consensus.cc:482] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.677114 24950 raft_consensus.cc:2454] T aca59ad77ad642c48a7e7eed1635089d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:15.616017 25025 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:15.651321 28243 raft_consensus.cc:2781] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240426 13:20:15.686774 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.537348 28339 consensus_queue.cc:260] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.584448 28055 raft_consensus.cc:504] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.659677 28152 leader_election.cc:288] T 3fe8f3015c01480fb67b39d3f5a947c7 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.708982 24947 raft_consensus.cc:2354] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 7. Current term is 8.
I20240426 13:20:15.667034 28237 leader_election.cc:288] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.712828 24921 leader_election.cc:302] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 1590f80bdade41c6a755b8501c55fe16, 30c2bb0adfa54bbfa3cb39773d4043c8
I20240426 13:20:15.770540 24951 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.771266 25620 raft_consensus.cc:1261] T 0fbdcc6d88d14934b64bc2a29382519f P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Refusing update from remote peer 46106ff992984d33bb4a4d8f30162639: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20240426 13:20:15.788213 24921 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.3:38089, user_credentials={real_user=slave}} blocked reactor thread for 75450.9us
I20240426 13:20:15.791088 24921 leader_election.cc:302] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e; no voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639
W20240426 13:20:15.708483 28284 consensus_meta.cc:287] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.740s user 0.003s sys 0.012s
I20240426 13:20:15.724144 25029 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 4.
I20240426 13:20:15.800019 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2212db2dbcf486888aefb7dd0174287" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.801358 25025 raft_consensus.cc:2454] T a2212db2dbcf486888aefb7dd0174287 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:15.672077 28279 raft_consensus.cc:504] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.714391 28326 raft_consensus.cc:2726] T 08f19147e9214114a8749e2a8196bacd P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240426 13:20:15.704443 28034 raft_consensus.cc:729] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.660220 28316 consensus_queue.cc:237] T 8f0f51d042f24e54b9e3a33e390f5982 P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.810308 25624 raft_consensus.cc:2354] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:20:15.815908 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.816797 24951 raft_consensus.cc:2379] T 9911e0422347463c8781b2574c023744 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 9: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
W20240426 13:20:15.802290 28352 log.cc:924] Time spent T 81d7dd33a3494a2d94627156d2fc1615 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.077s user 0.001s sys 0.001s
I20240426 13:20:15.818655 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.813565 24921 leader_election.cc:302] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:15.823124 25023 raft_consensus.cc:2454] T 242110e36ca34176b9d522087d440de6 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in term 9.
I20240426 13:20:15.789677 28049 raft_consensus.cc:482] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.798204 28171 raft_consensus.cc:2726] T c997ccc9a12742c3b149dc9747b2a0ae P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.845038 25025 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:15.851564 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb99666c521d49ada45165233068daf3" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.851773 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.790143 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d066b73b2e464d80b7b667e05602d879" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:15.857136 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.857442 25616 raft_consensus.cc:3037] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Advancing to term 7
I20240426 13:20:15.793649 27847 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:15.860327 25096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.860488 27847 raft_consensus.cc:504] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.861167 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9911e0422347463c8781b2574c023744" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.862473 27847 leader_election.cc:288] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:15.816010 28281 consensus_meta.cc:287] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e: Time spent flushing consensus metadata: real 0.757s user 0.010s sys 0.004s
I20240426 13:20:15.863153 28281 raft_consensus.cc:504] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:15.865082 28281 leader_election.cc:288] T 2d6f5c5283464089ad1bc18c13b427cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.866334 28243 raft_consensus.cc:482] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:15.866936 28243 raft_consensus.cc:3037] T 9108945c08ba4b2287c1e238b402722f P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.806115 28166 consensus_queue.cc:1225] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Peer 46106ff992984d33bb4a4d8f30162639 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20240426 13:20:15.856490 28351 raft_consensus.cc:2781] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election won for term 10
I20240426 13:20:15.868302 28351 raft_consensus.cc:686] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 LEADER]: Becoming Leader. State: Replica: 1e87e7bb414642aeb72fe0d2ff72cc2e, State: Running, Role: LEADER
I20240426 13:20:15.869027 28096 leader_election.cc:288] T 499552c0877e46c5af41389ff1be3152 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 10 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:15.871003 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.871392 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6324309bd7cf40778c508e740dcf979d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.801738 28190 consensus_queue.cc:1035] T a73f20bc0b224284bc0bf8eed62401d6 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240426 13:20:15.826046 28242 consensus_queue.cc:1035] T 0fbdcc6d88d14934b64bc2a29382519f P 46106ff992984d33bb4a4d8f30162639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20240426 13:20:15.883065 25100 raft_consensus.cc:2379] T aca63e93aec8413b83ec63e853c8f403 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:15.869235 28351 consensus_queue.cc:237] T 7f413a75d3c249dbb47c1ec61978c2ea P 1e87e7bb414642aeb72fe0d2ff72cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.788192 27923 raft_consensus.cc:504] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.898883 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.899284 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.810480 28305 consensus_queue.cc:1035] T 53343363b25040c29f4c212b440dc2cc P 1590f80bdade41c6a755b8501c55fe16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20240426 13:20:15.920944 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8fda82f95348219809212d99d6704e" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.921934 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4076b67a4fe4b3297158d094e10de08" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.849673 28295 raft_consensus.cc:482] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:15.801571 28164 consensus_queue.cc:237] T f49912f064d045d8a1d8be2c08b97d44 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.942370 25102 raft_consensus.cc:3037] T 0fbdcc6d88d14934b64bc2a29382519f P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Advancing to term 8
I20240426 13:20:15.899979 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:15.896827 24956 raft_consensus.cc:2396] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20240426 13:20:15.870098 28339 raft_consensus.cc:3037] T aca59ad77ad642c48a7e7eed1635089d P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 5 FOLLOWER]: Advancing to term 6
I20240426 13:20:15.931250 25025 raft_consensus.cc:2379] T 79bb183792fd4ae08dd4564ec82efc36 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 9: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:15.931483 24947 raft_consensus.cc:1226] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Rejecting Update request from peer 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4. Ops: []
I20240426 13:20:15.931572 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.931281 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04bd8996e6349879e25fca4e52dbda9" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:15.947727 25023 raft_consensus.cc:2454] T 99a876ca7b6e4082b58359403a47730d P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 10.
I20240426 13:20:15.948664 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.938057 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9108945c08ba4b2287c1e238b402722f" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.936996 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e6a71a0c24a3aa4f1348fd2ae9fe4" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:15.951228 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:15.952390 24951 raft_consensus.cc:2379] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 7: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:15.953671 25589 leader_election.cc:302] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:15.964867 25100 raft_consensus.cc:2379] T 009697c9dea4421d868c467f0cd6c079 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:15.949770 24948 raft_consensus.cc:2454] T 9108945c08ba4b2287c1e238b402722f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:15.923035 28340 raft_consensus.cc:3032] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Stepping down as leader of term 6
I20240426 13:20:15.972801 25024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:15.972999 25069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 244925us
I20240426 13:20:15.979359 25024 raft_consensus.cc:2396] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240426 13:20:15.924538 27498 raft_consensus.cc:3032] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Stepping down as leader of term 8
I20240426 13:20:15.980875 25024 raft_consensus.cc:1261] T 680b3a9e21d44d71987bc91ce797f547 P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240426 13:20:15.984508 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bb183792fd4ae08dd4564ec82efc36" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.985169 24948 raft_consensus.cc:2454] T 79bb183792fd4ae08dd4564ec82efc36 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.987339 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7408112bace482693c999f678aa9d73" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.873088 28049 raft_consensus.cc:504] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } attrs { replace: true } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:15.947036 28326 raft_consensus.cc:2726] T 5f4cb6239dd94f7693fdaacdc8ae8a8f P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240426 13:20:15.848453 28255 consensus_queue.cc:260] T 4010c37c00d94e78b5e3260ad78e7904 P 46106ff992984d33bb4a4d8f30162639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.019341 25025 raft_consensus.cc:1261] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: Refusing update from remote peer 30c2bb0adfa54bbfa3cb39773d4043c8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20240426 13:20:16.012387 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680b3a9e21d44d71987bc91ce797f547" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:15.947072 28025 leader_election.cc:288] T 009697c9dea4421d868c467f0cd6c079 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
W20240426 13:20:16.021770 24995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 185985us
I20240426 13:20:15.978678 28340 raft_consensus.cc:729] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:15.982338 27498 raft_consensus.cc:729] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 30c2bb0adfa54bbfa3cb39773d4043c8, State: Running, Role: LEADER
I20240426 13:20:16.019908 25025 pending_rounds.cc:76] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting all ops after (but not including) 4
I20240426 13:20:16.028170 25025 pending_rounds.cc:97] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20240426 13:20:16.026446 24948 raft_consensus.cc:2454] T 680b3a9e21d44d71987bc91ce797f547 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 8.
I20240426 13:20:15.989573 27738 raft_consensus.cc:2781] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20240426 13:20:16.003672 28325 consensus_meta.cc:287] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16: Time spent flushing consensus metadata: real 0.631s user 0.010s sys 0.001s
I20240426 13:20:16.001704 28161 raft_consensus.cc:482] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.036005 25023 raft_consensus.cc:2396] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20240426 13:20:16.028455 25025 raft_consensus.cc:2866] T ece98ef0f07f4184bedf2cb96b628f20 P 46106ff992984d33bb4a4d8f30162639 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240426 13:20:16.009272 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.038132 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece98ef0f07f4184bedf2cb96b628f20" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.038861 24948 raft_consensus.cc:2454] T ece98ef0f07f4184bedf2cb96b628f20 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:16.041937 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a876ca7b6e4082b58359403a47730d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.035065 24995 leader_election.cc:398] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Vote denied by peer 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069) with higher term. Message: Invalid argument: T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 for earlier term 3. Current term is 4.
I20240426 13:20:15.947170 28295 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.044382 24995 leader_election.cc:401] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240426 13:20:16.051064 28025 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240426 13:20:16.051409 28062 consensus_meta.cc:287] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8: Time spent flushing consensus metadata: real 0.527s user 0.006s sys 0.005s
I20240426 13:20:16.051846 28025 raft_consensus.cc:504] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.052390 28062 raft_consensus.cc:504] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.055955 25101 raft_consensus.cc:595] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Received request to transfer leadership to TS 46106ff992984d33bb4a4d8f30162639
I20240426 13:20:16.056493 25101 raft_consensus.cc:603] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240426 13:20:16.056977 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.057878 28025 leader_election.cc:288] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.059531 25100 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:16.061252 24921 leader_election.cc:302] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 30c2bb0adfa54bbfa3cb39773d4043c8; no voters:
I20240426 13:20:16.062516 28025 raft_consensus.cc:2781] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240426 13:20:16.064445 28171 raft_consensus.cc:482] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.065091 28171 raft_consensus.cc:504] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.063390 28025 raft_consensus.cc:482] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240426 13:20:16.069841 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca59ad77ad642c48a7e7eed1635089d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:15.937175 28034 consensus_queue.cc:260] T 9911e0422347463c8781b2574c023744 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.036350 28365 log.cc:924] Time spent T f533a7c13bae41539cdc0a425b282e53 P 30c2bb0adfa54bbfa3cb39773d4043c8: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20240426 13:20:16.078006 25023 raft_consensus.cc:2379] T aca59ad77ad642c48a7e7eed1635089d P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 6: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:16.034901 27738 raft_consensus.cc:482] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 30c2bb0adfa54bbfa3cb39773d4043c8)
I20240426 13:20:16.127818 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.080400 25025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73f20bc0b224284bc0bf8eed62401d6" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
W20240426 13:20:16.209424 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.2:43671, user_credentials={real_user=slave}} blocked reactor thread for 82344.9us
I20240426 13:20:16.121461 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.211284 24948 raft_consensus.cc:2379] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 46106ff992984d33bb4a4d8f30162639 in current term 3: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.118000 25097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be16e3d2b66a44c5ac4a072b891540a2" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.125418 24955 raft_consensus.cc:1261] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20240426 13:20:16.120852 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.213765 24956 raft_consensus.cc:2379] T 786abf3f43724ad5b959e497fc776d1d P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 11: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.121735 24952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9910a5336aa4e3cab11dca4a556fabf" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.074976 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.178676 25098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8"
I20240426 13:20:16.118889 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f51d042f24e54b9e3a33e390f5982" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.218320 24949 raft_consensus.cc:2379] T 8f0f51d042f24e54b9e3a33e390f5982 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 6: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.121157 24950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28ef3032ef44d2d89c750a8d4892f00" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.115980 25616 raft_consensus.cc:2454] T d066b73b2e464d80b7b667e05602d879 P 1590f80bdade41c6a755b8501c55fe16 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 7.
I20240426 13:20:16.039893 28242 raft_consensus.cc:482] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.215194 24952 raft_consensus.cc:3037] T b9910a5336aa4e3cab11dca4a556fabf P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:16.123795 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.120234 24951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe8f3015c01480fb67b39d3f5a947c7" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.229759 25023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.230146 24951 raft_consensus.cc:2454] T 3fe8f3015c01480fb67b39d3f5a947c7 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 9.
I20240426 13:20:16.229782 25099 raft_consensus.cc:2379] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:16.221210 24950 raft_consensus.cc:2379] T a28ef3032ef44d2d89c750a8d4892f00 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 9: Already voted for candidate 30c2bb0adfa54bbfa3cb39773d4043c8 in this term.
I20240426 13:20:16.120536 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6277899576264e68aa43514e5b5c676e" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.222630 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" is_pre_election: true
I20240426 13:20:16.179360 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fad756437243d2959f5998d6bf8309" candidate_uuid: "46106ff992984d33bb4a4d8f30162639" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16" is_pre_election: true
W20240426 13:20:16.110478 28358 log.cc:924] Time spent T f7408112bace482693c999f678aa9d73 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20240426 13:20:16.109815 28363 log.cc:924] Time spent T 718dfae59b1148c2b6cc522821ea4ea4 P 46106ff992984d33bb4a4d8f30162639: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20240426 13:20:16.245074 24947 raft_consensus.cc:2454] T 6277899576264e68aa43514e5b5c676e P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 3.
I20240426 13:20:16.067546 28171 leader_election.cc:288] T 81d7dd33a3494a2d94627156d2fc1615 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.116946 27923 leader_election.cc:288] T be16e3d2b66a44c5ac4a072b891540a2 P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.220628 27738 raft_consensus.cc:3037] T 99a876ca7b6e4082b58359403a47730d P 1590f80bdade41c6a755b8501c55fe16 [term 10 FOLLOWER]: Advancing to term 11
I20240426 13:20:16.249013 24953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256415e42b6c444387115ab766a5a16c" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.249317 24949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" is_pre_election: true
I20240426 13:20:16.251438 25070 leader_election.cc:302] T 9108945c08ba4b2287c1e238b402722f P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30c2bb0adfa54bbfa3cb39773d4043c8, 46106ff992984d33bb4a4d8f30162639; no voters: 1e87e7bb414642aeb72fe0d2ff72cc2e
I20240426 13:20:16.079627 28025 raft_consensus.cc:3037] T 551154b9c3494e4da288affc4b88c361 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Advancing to term 5
I20240426 13:20:16.065238 28062 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 30c2bb0adfa54bbfa3cb39773d4043c8 [CANDIDATE]: Term 9 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.094867 27911 consensus_queue.cc:1035] T ece98ef0f07f4184bedf2cb96b628f20 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20240426 13:20:16.109033 28049 leader_election.cc:288] T 11fad756437243d2959f5998d6bf8309 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.122601 28295 leader_election.cc:288] T 764a1a5c07f94909b2fd69a50de4bedd P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.119050 28281 raft_consensus.cc:482] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240426 13:20:16.296566 28281 raft_consensus.cc:504] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.298962 28281 leader_election.cc:288] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 1590f80bdade41c6a755b8501c55fe16 (127.23.61.4:40069)
I20240426 13:20:16.117178 27498 consensus_queue.cc:260] T 79bb183792fd4ae08dd4564ec82efc36 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.300248 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cd5654ae4294fcc8482e45f83032773" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.301424 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764a1a5c07f94909b2fd69a50de4bedd" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.083828 28055 leader_election.cc:288] T 6277899576264e68aa43514e5b5c676e P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 4 election: Requested vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671)
I20240426 13:20:16.046111 28161 raft_consensus.cc:504] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
I20240426 13:20:16.305613 24951 raft_consensus.cc:1261] T 59632e528afa44ca98a626355e1e4660 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 8 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240426 13:20:16.306515 28161 leader_election.cc:288] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.307552 25025 raft_consensus.cc:2454] T a73f20bc0b224284bc0bf8eed62401d6 P 46106ff992984d33bb4a4d8f30162639 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 10.
I20240426 13:20:16.115264 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca63e93aec8413b83ec63e853c8f403" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
W20240426 13:20:16.311790 25031 consensus_meta.cc:287] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639: Time spent flushing consensus metadata: real 0.630s user 0.009s sys 0.000s
I20240426 13:20:16.312242 25031 raft_consensus.cc:2454] T d3a669ffe77a4ad29604f4ecd7c2644a P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:16.313333 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6f5c5283464089ad1bc18c13b427cc" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.314334 25031 raft_consensus.cc:2379] T 2d6f5c5283464089ad1bc18c13b427cc P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in current term 7: Already voted for candidate 46106ff992984d33bb4a4d8f30162639 in this term.
I20240426 13:20:16.315181 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009697c9dea4421d868c467f0cd6c079" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.308954 24950 raft_consensus.cc:1226] T f4076b67a4fe4b3297158d094e10de08 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 6 FOLLOWER]: Rejecting Update request from peer 1590f80bdade41c6a755b8501c55fe16 for earlier term 5. Current term is 6. Ops: []
I20240426 13:20:16.249083 28164 consensus_queue.cc:1046] T d066b73b2e464d80b7b667e05602d879 P 30c2bb0adfa54bbfa3cb39773d4043c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20240426 13:20:16.244796 28242 raft_consensus.cc:504] T 786abf3f43724ad5b959e497fc776d1d P 46106ff992984d33bb4a4d8f30162639 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
W20240426 13:20:16.321686 25588 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.61.1:41817, user_credentials={real_user=slave}} blocked reactor thread for 244743us
W20240426 13:20:16.323457 25588 leader_election.cc:334] T 786abf3f43724ad5b959e497fc776d1d P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1e87e7bb414642aeb72fe0d2ff72cc2e (127.23.61.1:41817): Timed out: RequestConsensusVote RPC to 127.23.61.1:41817 timed out after 1.796s (SENT)
I20240426 13:20:16.285269 24949 raft_consensus.cc:2379] T 3014c46944bc4cbb82624ac909221634 P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1590f80bdade41c6a755b8501c55fe16 in current term 4: Already voted for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in this term.
I20240426 13:20:16.324594 25616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3014c46944bc4cbb82624ac909221634" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
I20240426 13:20:16.321728 25589 leader_election.cc:302] T d3a669ffe77a4ad29604f4ecd7c2644a P 1590f80bdade41c6a755b8501c55fe16 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590f80bdade41c6a755b8501c55fe16, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:16.327093 25615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786abf3f43724ad5b959e497fc776d1d" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "1590f80bdade41c6a755b8501c55fe16"
W20240426 13:20:16.260015 28349 log.cc:924] Time spent T e915e90a1a6847899575125ef7e68f2a P 1590f80bdade41c6a755b8501c55fe16: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240426 13:20:16.096122 28206 raft_consensus.cc:3037] T 6277899576264e68aa43514e5b5c676e P 46106ff992984d33bb4a4d8f30162639 [term 3 FOLLOWER]: Advancing to term 4
I20240426 13:20:16.360819 25026 raft_consensus.cc:2454] T aca63e93aec8413b83ec63e853c8f403 P 46106ff992984d33bb4a4d8f30162639 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 6.
I20240426 13:20:16.040928 28325 raft_consensus.cc:504] T c896b58b7766401eac0a7f69574827f1 P 1590f80bdade41c6a755b8501c55fe16 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } } peers { permanent_uuid: "1590f80bdade41c6a755b8501c55fe16" member_type: VOTER last_known_addr { host: "127.23.61.4" port: 40069 } attrs { promote: false } }
I20240426 13:20:16.289724 24947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a669ffe77a4ad29604f4ecd7c2644a" candidate_uuid: "1590f80bdade41c6a755b8501c55fe16" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e"
I20240426 13:20:16.303234 24954 raft_consensus.cc:1261] T 53343363b25040c29f4c212b440dc2cc P 1e87e7bb414642aeb72fe0d2ff72cc2e [term 10 FOLLOWER]: Refusing update from remote peer 1590f80bdade41c6a755b8501c55fe16: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20240426 13:20:16.302033 25027 raft_consensus.cc:2454] T 764a1a5c07f94909b2fd69a50de4bedd P 46106ff992984d33bb4a4d8f30162639 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1590f80bdade41c6a755b8501c55fe16 in term 8.
I20240426 13:20:16.302376 25620 raft_consensus.cc:2454] T 11fad756437243d2959f5998d6bf8309 P 1590f80bdade41c6a755b8501c55fe16 [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 46106ff992984d33bb4a4d8f30162639 in term 6.
I20240426 13:20:16.315896 25031 raft_consensus.cc:2454] T 009697c9dea4421d868c467f0cd6c079 P 46106ff992984d33bb4a4d8f30162639 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 7.
I20240426 13:20:16.367976 25027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499552c0877e46c5af41389ff1be3152" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639"
I20240426 13:20:16.368189 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551154b9c3494e4da288affc4b88c361" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.368834 25031 raft_consensus.cc:2454] T 551154b9c3494e4da288affc4b88c361 P 46106ff992984d33bb4a4d8f30162639 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e87e7bb414642aeb72fe0d2ff72cc2e in term 4.
I20240426 13:20:16.369593 25031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81d7dd33a3494a2d94627156d2fc1615" candidate_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "46106ff992984d33bb4a4d8f30162639" is_pre_election: true
I20240426 13:20:16.128572 28279 leader_election.cc:288] T 3cd5654ae4294fcc8482e45f83032773 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 8 election: Requested vote from peers 46106ff992984d33bb4a4d8f30162639 (127.23.61.2:43671), 30c2bb0adfa54bbfa3cb39773d4043c8 (127.23.61.3:38089)
I20240426 13:20:16.135538 24921 leader_election.cc:302] T a2212db2dbcf486888aefb7dd0174287 P 1e87e7bb414642aeb72fe0d2ff72cc2e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e87e7bb414642aeb72fe0d2ff72cc2e, 46106ff992984d33bb4a4d8f30162639; no voters:
I20240426 13:20:16.195591 28340 consensus_queue.cc:260] T 861e6a71a0c24a3aa4f1348fd2ae9fe4 P 30c2bb0adfa54bbfa3cb39773d4043c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e87e7bb414642aeb72fe0d2ff72cc2e" member_type: VOTER last_known_addr { host: "127.23.61.1" port: 41817 } } peers { permanent_uuid: "46106ff992984d33bb4a4d8f30162639" member_type: VOTER last_known_addr { host: "127.23.61.2" port: 43671 } } peers { permanent_uuid: "30c2bb0adfa54bbfa3cb39773d4043c8" member_type: VOTER last_known_addr { host: "127.23.61.3" port: 38089 } }
W20240426 13:20:16.350277 25070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote ->